Skip to content

Rounding and Significant Figures

The Story Behind the Math

In 1940, aeronautical engineers at the National Advisory Committee for Aeronautics (NACA, precursor to NASA) encountered a disaster. A critical calculation for wing design was off by 10%—not because the math was wrong, but because engineers rounded intermediate results too aggressively.

The problem: Rounding errors compound. A small error in step 1 becomes a larger error in step 10.

This led to the formalization of significant figures—rules for tracking precision through calculations. During WWII, similar issues plagued artillery calculations, ballistics tables, and atomic research. Precision wasn't just academic—it was life or death.

The modern irony: Computers calculate with extreme precision, but humans still need to decide what to report. Should that be 3.14 or 3.14159? The answer depends on what you know, not what your calculator shows.

Why It Matters

Rounding and significant figures affect:

  • Science: Reporting measurements with appropriate precision
  • Engineering: Ensuring calculations maintain required accuracy
  • Finance: Currency calculations, exchange rates, interest
  • Medicine: Drug dosages, test results, treatment protocols
  • Computing: Floating-point arithmetic, numerical stability
  • Everyday life: Estimating costs, distances, quantities

Understanding precision prevents false confidence in numbers and catches calculation errors.

Prerequisites

The Core Insight

Why We Round

Measurements are never exact. A ruler marked in millimeters can't measure micrometers. Rounding reflects the limit of our knowledge.

If you measure a table as 1.24 meters using a tape measure marked in centimeters: - You know it's about 1.24 meters - You don't know if it's 1.241 or 1.239 meters - Reporting 1.2400 would be dishonest—you don't have that precision

The Fundamental Tension

  • More digits: Implies more precision (may be misleading)
  • Fewer digits: Loses information (may be insufficient)

Significant figures are the compromise: report all digits you know, and no more.

Rounding Rules

The Basic Rule

To round to a specific place: 1. Identify the digit at the target place 2. Look at the digit immediately to its right 3. If that digit is 5 or greater, round up the target digit 4. If less than 5, keep the target digit unchanged 5. Replace all digits to the right with zeros (or drop them after decimal)

Examples

Round 3.14159 to: - 2 decimal places: 3.14 (next digit is 1 < 5) - 3 decimal places: 3.142 (next digit is 5 ≥ 5, round up) - 1 decimal place: 3.1 (next digit is 4 < 5) - Whole number: 3 (next digit is 1 < 5)

Round 2,847 to: - Nearest 10: 2,850 (4 < 5, but we need the 0 placeholder) - Nearest 100: 2,800 - Nearest 1,000: 3,000

The "5" Rule (Banker's Rounding)

What about exactly 5? Standard rule: round up.

But this introduces bias—over many calculations, you'll round up slightly more often.

Banker's rounding (IEEE 754 standard): - Round to nearest even digit when exactly halfway - 2.5 → 2, 3.5 → 4

This balances rounding up and down over time.

Significant Figures

What Are Significant Figures?

Significant figures (sig figs) are the digits in a number that carry meaning about its precision.

Rules for counting significant figures:

  1. Non-zero digits are always significant: 345 has 3 sig figs
  2. Zeros between non-zero digits are significant: 3.05 has 3 sig figs
  3. Leading zeros are NOT significant: 0.0045 has 2 sig figs
  4. Trailing zeros after decimal ARE significant: 3.400 has 4 sig figs
  5. Trailing zeros without decimal are ambiguous: 3400 could be 2, 3, or 4 sig figs

Examples

Number Significant Figures Explanation
123 3 All non-zero
0.0123 3 Leading zeros not significant
100.5 4 Zeros between non-zeros are significant
100 Ambiguous Could be 1, 2, or 3
100. 3 Decimal indicates precision
1.00 × 10² 3 Scientific notation makes it clear

Scientific Notation Removes Ambiguity

Instead of writing 3400 (ambiguous), write: - \(3.4 \times 10^3\) = 2 sig figs - \(3.40 \times 10^3\) = 3 sig figs
- \(3.400 \times 10^3\) = 4 sig figs

The mantissa (the part before × 10ⁿ) always shows significant figures clearly.

Operations with Significant Figures

Multiplication and Division

The result has the same number of significant figures as the factor with the fewest sig figs.

Example: \(4.56 \times 1.4\)

\[ 4.56 \times 1.4 = 6.384 \]
  • 4.56 has 3 sig figs
  • 1.4 has 2 sig figs
  • Result should have 2 sig figs: 6.4

Why? The least precise measurement limits the precision of the result. If you only know one factor to 2 sig figs, you can't magically gain precision by calculating.

Example: \(100 \div 3.0\)

\[ 100 \div 3.0 = 33.333\ldots \]
  • 100 (ambiguous, assume 1 sig fig)
  • 3.0 has 2 sig figs
  • Result should have 1 sig fig: 30

Addition and Subtraction

The result has the same number of decimal places as the number with the fewest decimal places.

Example: \(12.345 + 1.2\)

\[ 12.345 + 1.2 = 13.545 \]
  • 12.345 has 3 decimal places
  • 1.2 has 1 decimal place
  • Result should have 1 decimal place: 13.5

Why the different rule? With addition, decimal place matters, not total sig figs. You can't know the thousandths place if one number is only precise to tenths.

Example: \(1500 + 24.5\)

\[ 1500 + 24.5 = 1524.5 \]
  • 1500 (ambiguous precision, assume ones place)
  • 24.5 has tenths place
  • Result should be rounded to ones: 1525

Combined Operations

When mixing operations, track significant figures at each step, or keep extra digits during calculation and round only at the end.

Example: \((4.5 \times 3.12) + 2.1\)

Step 1: \(4.5 \times 3.12 = 14.04\) → round to 2 sig figs: 14

Step 2: \(14 + 2.1 = 16.1\) → round to ones place: 16

Or with extra precision: \(14.04 + 2.1 = 16.14\) → round to 16

The Mathematical Foundation

Error Propagation

When we write \(3.14\), we mean "somewhere between 3.135 and 3.145."

The range of uncertainty: - \(3.14 \pm 0.005\) (2 decimal places) - This represents about 0.16% uncertainty

Multiplication error: If \(x\) has uncertainty \(\Delta x\) and \(y\) has uncertainty \(\Delta y\):

\[ \frac{\Delta(xy)}{xy} \approx \frac{\Delta x}{x} + \frac{\Delta y}{y} \]

The relative errors add. This is why sig figs multiply the way they do.

Logarithms and Significant Figures

In logarithms, the number of decimal places in the result equals the number of sig figs in the input:

\[ \log_{10}(3.0 \times 10^5) = 5.48 \]

3.0 has 2 sig figs, so the answer has 2 decimal places.

Exact Numbers vs. Measurements

Exact Numbers

Some numbers have infinite precision: - Counted quantities: 12 eggs, 5 students - Defined conversions: 1 inch = 2.54 cm (exactly) - Mathematical constants in formulas: \(\pi\), \(e\)

Exact numbers don't limit significant figures.

Example: Calculate circumference of circle with radius \(r = 2.5\) cm

\[ C = 2\pi r = 2 \times \pi \times 2.5 = 15.707\ldots \]
  • \(2\) is exact
  • \(\pi\) is a mathematical constant (use as many digits as needed)
  • \(2.5\) has 2 sig figs
  • Result: 16 cm (2 sig figs)

Practical Applications

Scientific Reporting

Wrong: "The reaction took 45.23847 seconds"

Right: "The reaction took 45 seconds" (if measured with a stopwatch)

Only report digits you actually measured.

Engineering Tolerances

A specification says: "Length: \(100 \pm 0.5\) mm"

This means: - Acceptable range: 99.5 mm to 100.5 mm - Measurement should be reported to tenths place - 100.0 mm implies ±0.05 mm precision (too precise!)

Financial Calculations

Currency is often rounded to cents (2 decimal places), but:

  • Intermediate calculations should keep full precision
  • Only round the final result
  • Rounding each step creates cumulative errors

Example: Three items at $1.333 each

Step-by-step rounding: $1.33 + $1.33 + $1.33 = $3.99 Exact: $3.999 → $4.00

Computing

Floating-point precision: Computers use binary floating point with about 15-16 decimal digits of precision.

Catastrophic cancellation: Subtracting nearly equal numbers loses precision:

\[ 1.0000001 - 1.0000000 = 0.0000001 \]

Both numbers had 8 sig figs, but the result has only 1!

Common Misconceptions

  1. "More decimal places = more accurate": No! Extra digits from calculator don't add precision.

  2. "Round at each step": Don't! Round only at the end to avoid error accumulation.

  3. "Trailing zeros are never significant": They are after a decimal point (3.400 is more precise than 3.4).

  4. "Exact conversion factors limit precision": Defined conversions (1 inch = 2.54 cm) are exact—they don't limit sig figs.

  5. "All measured digits are significant": Not leading zeros. 0.0045 has 2 sig figs, not 4.

Visual Understanding

The Precision Ladder

Most Precise                    Least Precise
     ↓                               ↓
3.14159  →  3.1416  →  3.142  →  3.14  →  3.1  →  3
  (6 sf)      (5 sf)      (4 sf)     (3 sf)    (2 sf)  (1 sf)

Each step loses information but gains clarity.

Measurement Precision

Ruler marked in millimeters (mm):

|----|----|----|----|
0    10   20   30   mm

You can estimate to nearest 0.5 mm, so measurements should be reported as: - 23.5 mm (3 sig figs) - Not 23.500 mm (you don't have that precision)

Best Practices

For Students

  1. Keep extra digits during calculation, round at the end
  2. Use scientific notation to avoid ambiguity
  3. Understand your measuring tool's precision
  4. Report uncertainty when possible (\(23.5 \pm 0.5\) mm)

For Scientists and Engineers

  1. Always state precision explicitly: "3.14 (3 sig figs)"
  2. Propagate errors properly for critical calculations
  3. Consider logarithms for multiplicative processes
  4. Use interval arithmetic for safety-critical systems

References

  • Taylor, J. R. (1997). An Introduction to Error Analysis: The Study of Uncertainties in Physical Measurements (2nd ed.). University Science Books.
  • ISO 80000-1:2009. Quantities and units—Part 1: General.
  • Higham, N. J. (2002). Accuracy and Stability of Numerical Algorithms (2nd ed.). SIAM.