Electronics Guide

Binary Floating-Point Formats

IEEE 754 defines binary formats of 16, 32, 64, and 128 bits. Each consists of a sign bit, a biased exponent field, and a trailing significand field that stores the fraction bits of the significand, which older texts call the mantissa. The precision p counts the stored bits plus one implicit leading bit.

The decimal-digit column is p × log₁₀(2). Subnormal numbers extend each range below the smallest normal value, down to 5.96 × 10⁻⁸ in binary16, 1.40 × 10⁻⁴⁵ in binary32, 4.94 × 10⁻³²⁴ in binary64, and 6.48 × 10⁻⁴⁹⁶⁶ in binary128.

The standard designates binary32, binary64, and binary128 as basic formats. Binary16 is defined as an interchange format, intended chiefly for storage, although an implementation may also compute in it, and many GPUs and recent processors do. The standard also describes extended precision formats wider than a basic format. The best known is the 80-bit double-extended format of the x87 floating-point unit, descended from the Intel 8087, which pairs a 15-bit exponent with a 64-bit significand that stores its leading bit explicitly, giving about 19 decimal digits.

Representation Details

In a normalized number the significand is at least 1 and less than 2, so its leading bit is always 1. That bit is implicit rather than stored, which gains a bit of precision. With S the sign bit, E the stored exponent field read as an unsigned integer, and F the stored fraction bits, a normalized number has the value

value = (−1)^S × 1.F × 2^{E − bias}

The bias lets an unsigned exponent field represent exponents both above and below zero. For a k-bit exponent field the bias is 2^k−1 − 1: 15 for binary16, 127 for binary32, 1023 for binary64, and 16,383 for binary128. The all-zeros and all-ones exponent fields are reserved for zeros, subnormals, infinities, and NaNs, so normal exponents run from e_min = 1 − bias to e_max = bias, which is −126 to +127 in binary32. Because the biased exponent sits above the fraction, the bit patterns of non-negative floating-point numbers order the same way as the unsigned integers they resemble, which lets hardware compare two such values with an ordinary integer comparison.

For example, the binary32 bit pattern 0x40490FDB has S = 0 and E = 128, so its unbiased exponent is 1, and its fraction bits give a significand of about 1.5707964. Its value is therefore about 1.5707964 × 2¹ = 3.14159274, the binary32 number nearest π, which it exceeds by about 8.7 × 10⁻⁸.

Special Values

IEEE 754 defines several special values to handle exceptional cases:

• Zero: An exponent field and a fraction of all zeros. Both +0 and −0 exist. They compare equal, but some operations distinguish them: 1/(+0) = +∞, while 1/(−0) = −∞.
• Infinity: The all-ones exponent field with a zero fraction. Positive and negative infinity are the default results of overflow under round to nearest and of dividing a nonzero finite number by zero, and they behave as limits in arithmetic, so ∞ + 1 = ∞.
• NaN (Not a Number): The all-ones exponent field with a nonzero fraction. A NaN is the default result of an invalid operation such as 0/0, ∞ − ∞, or the square root of a negative number. Quiet NaNs propagate through most operations without raising exceptions, while signaling NaNs raise the invalid-operation exception when used as operands. The 2008 revision recommends that the leading fraction bit be 1 in a quiet NaN and 0 in a signaling one. A NaN compares unordered with every value, itself included, so the comparison x = x is false only when x is a NaN.

Purpose and Representation

When the exponent field is zero and the fraction is nonzero, the number is subnormal. The implicit leading bit becomes 0 instead of 1, and the exponent is fixed at the minimum normal exponent, e_min = 1 − bias:

value = (−1)^S × 0.F × 2^{1 − bias}

The subnormals are evenly spaced, with the same spacing as the normal numbers in the lowest binade, the interval from the smallest normal number to twice that value. They therefore fill the gap between zero and the smallest normal number without a jump, which guarantees that x − y = 0 if and only if x = y for any two finite floating-point numbers x and y. Each subnormal carries fewer significant bits than a normal number, down to a single bit for the smallest, so relative precision degrades as values approach zero.

Hardware Implications

Subnormal numbers present significant challenges for hardware implementation:

• Variable leading zeros: A subnormal significand may have any number of leading zeros, so a subnormal operand needs a variable normalization shift before multiplication or division, and a result that falls into the subnormal range must be shifted right before it is rounded.
• Performance penalty: Many designs handle these cases outside the fast datapath, through microcode assists or traps to software. Agner Fog's measurements of several generations of Intel cores show a penalty of roughly 120 to 150 clock cycles whenever an operation on normal operands produces a subnormal result, against a few cycles for an ordinary addition or multiplication, while AMD's Zen cores take only a few extra cycles.
• Flush-to-zero mode: Replaces a subnormal result with zero, avoiding the penalty at the cost of gradual underflow.
• Denormals-are-zero mode: Treats subnormal operands as zero. Neither mode conforms to IEEE 754. On x86 processors both are controlled by bits in the MXCSR register, and the penalties disappear only when both are set.

Numerical Significance

Subnormal numbers maintain important mathematical properties:

• Gradual underflow: Results approach zero smoothly rather than jumping discontinuously.
• Subtraction reliability: The difference of two distinct finite numbers is never zero, so a program that checks x ≠ y before dividing by x − y cannot divide by zero. With flush-to-zero enabled, that guarantee is lost.
• Algorithm stability: Many numerical algorithms behave more predictably with gradual underflow.

Round to Nearest, Ties to Even

The default attribute for binary formats returns the representable value nearest the exact result. When the exact result lies exactly halfway between two representable values, it returns the one whose least significant bit is 0, the "even" value. In binary32, for example, 16,777,217 (2²⁴ + 1) lies halfway between the representable neighbors 16,777,216 and 16,777,218, and it rounds to 16,777,216. This approach:

• Bounds each rounding error by half a unit in the last place, the smallest possible maximum error
• Avoids the statistical bias that always rounding halfway cases upward would accumulate over repeated operations
• Is the default in mainstream hardware and programming languages

Round to Nearest, Ties Away from Zero

Added in IEEE 754-2008, this attribute also rounds to the nearest value but breaks ties by rounding away from zero, the familiar schoolbook rule. The standard requires decimal implementations to provide it and does not require it for binary formats. Even for decimal formats, the standard recommends ties to even as the default.

Directed Rounding Modes

Three directed rounding modes always round in a specified direction:

• Round toward positive infinity (ceiling): Always rounds upward toward more positive values.
• Round toward negative infinity (floor): Always rounds downward toward more negative values.
• Round toward zero (truncation): Always rounds toward zero, discarding the bits beyond the destination precision.

These modes are essential for interval arithmetic, where computing upper and lower bounds of results enables rigorous error analysis.

Hardware Implementation

Rounding logic examines three bits beyond the destination precision:

• Guard bit: The first bit beyond the destination precision
• Round bit: The second bit beyond the destination precision
• Sticky bit: The logical OR of all remaining bits

Two positions are kept rather than one because normalization after a subtraction may shift the result left by one place, carrying the guard bit into the result. Once the result is normalized, the decision under round to nearest, ties to even is simple: increment the retained significand when the first discarded bit is 1 and either some later discarded bit is 1 or the retained least significant bit is 1. The directed attributes need only the sign and whether any discarded bit is nonzero. An increment can carry out of the significand, as when a significand of all ones rounds up, which forces a one-place renormalization and an exponent increment.

Exception Types

• Invalid operation: Signaled when an operation has no mathematically defined result, such as 0/0, 0 × ∞, ∞ − ∞, or √(−1), and by operations on signaling NaNs. The default result is a quiet NaN.
• Division by zero: Signaled when an operation on finite operands produces an exact infinite result, such as a nonzero number divided by zero or the logarithm of zero. The default result is a correctly signed infinity.
• Overflow: Signaled when the rounded result, computed as if the exponent range were unbounded, would exceed the largest finite number. The default result is an infinity or the largest finite number, with the sign of the exact result, depending on the rounding attribute; under round to nearest it is always an infinity.
• Underflow: Signaled when a nonzero result is tiny, smaller in magnitude than the smallest normal number. Under default handling the flag is raised only when the tiny result is also inexact, and the delivered result is the rounded subnormal or zero. For binary formats, the implementation chooses whether tininess is detected before or after rounding.
• Inexact: Signaled when the rounded result differs from the exact result. It is by far the most frequently raised exception, because most arithmetic results must be rounded, and programs rarely treat it as an error.

Status Flags

Each exception raises a corresponding status flag, which stays raised until the program explicitly lowers it. Because the flags accumulate, a program can run a long sequence of operations and test the flags once at the end rather than after every operation. C exposes them through the fetestexcept and feclearexcept functions of <fenv.h>.

Trap Handlers

When trapping is enabled for an exception, a handler receives control with information about the operation, its operands, and in some cases a tentative result. The 1985 standard specified, for example, that an overflow or underflow trap handler receive the result with its exponent wrapped back into range, so that the handler can extend the exponent range in software. A handler may:

• Substitute a different result value
• Log the exception for debugging
• Terminate the program
• Implement extended precision or range

Most general-purpose applications rely on default exception handling, but trap handlers are valuable for debugging and specialized numerical applications.

Accuracy Benefits

FMA eliminates the intermediate rounding error that would occur if the multiplication and addition were performed separately:

• Single rounding: The exact double-width product feeds the addition and is rounded only once, eliminating the intermediate rounding of a separate multiplication; under cancellation the improvement can far exceed one ulp (unit in the last place).
• Error-free products: If p is the rounded product of a and b, a single FMA computes (a × b) − p exactly, barring underflow, and so recovers the rounding error of the multiplication. Compensated summation and dot-product algorithms, such as those Ogita, Rump, and Oishi published in 2005, build on this to return results about as accurate as if they had been computed in twice the working precision.
• Correctly rounded division and square root: Newton-Raphson iterations for the reciprocal and the reciprocal square root converge quadratically with or without FMA. What FMA adds is an exact residual: for a candidate quotient q of a/b, one FMA computes the remainder a − (b × q) exactly, and its sign shows whether q or its neighbor is the correctly rounded quotient. Peter Markstein developed this approach for the IBM RISC System/6000, and it lets a processor without a hardware divider, such as Intel's Itanium, still deliver correctly rounded division and square root.
• Polynomial evaluation: Horner's method maps each step onto one FMA, and the single rounding per step improves accuracy in polynomial approximations.

Performance Advantages

FMA provides significant performance improvements:

• Reduced operation count: Many algorithms, from dot products to polynomial evaluation, are built from multiply-add pairs, and FMA halves their instruction count.
• Higher peak throughput: Each FMA counts as two floating-point operations, so a unit that issues one FMA per cycle doubles the peak rate of one that issues a multiply or an add.
• Better pipelining: Fewer instructions mean fewer dependencies to track and a shorter critical path through a chain of multiply-adds.
• Energy efficiency: Fusing the operations saves an instruction fetch and decode, a register-file write and read, and a separate normalization and rounding step.

Hardware Implementation

FMA hardware is more complex than separate multiplier and adder units:

• Wide datapath: The multiplier's 2p-bit product is kept unrounded, usually in carry-save form, until it meets the addend.
• Alignment shifter: The addend is shifted relative to the product within a window of about 3p + 2 bits, 161 bits for binary64. Designs place the unshifted addend just above the product's most significant bit and shift it right by an amount computed from the exponents, in parallel with the multiplication. An addend shifted past the window's lower end contributes only to the sticky bit.
• Adder: The sum spans the whole window, but the product occupies only its low 2p bits. A carry-save adder merges the product with the low part of the aligned addend, a 2p-bit carry-propagate adder completes the sum, and the high p + 2 bits, where only the addend lies, need just an incrementer to absorb a carry.
• Leading-zero anticipation: When the addend nearly cancels the product, the sum can have many leading zeros, so a leading-zero anticipator predicts the normalization shift in parallel with the addition instead of waiting for the sum.
• Combined normalization: A single normalization and rounding step serves both the multiplication and the addition.

IEEE 754-2008 Requirements

The 2008 revision of IEEE 754 made fusedMultiplyAdd a required operation for conforming implementations, and the 2019 revision retained it. Because the operation is defined as correctly rounded, algorithms designed around it produce identical results on every conforming platform, whether the FMA runs in hardware or in software emulation.

A fused result can differ from the separately rounded one, so a compiler may contract a multiplication and an addition into an FMA only where the language permits it; C controls this with the FP_CONTRACT pragma. Contraction can surprise: with fusion, a × b − a × b can evaluate to a nonzero value, because one product is rounded and the other is not.

Basic Architecture

A typical FPU contains several functional units:

• Adder/subtractor: Handles addition, subtraction, and comparison operations.
• Multiplier: Performs multiplication and often FMA operations.
• Divider: Executes division and sometimes square root using iterative algorithms.
• Conversion unit: Converts between integer and floating-point formats, and between different floating-point precisions.
• Special function unit: Evaluates transcendental functions in hardware in some designs, such as the sine and logarithm instructions of the x87 and the special function units of GPUs.

Pipeline Design

FPU operations are deeply pipelined to achieve high throughput:

• Addition pipeline: Typically three to five stages for exponent comparison, alignment, addition, normalization, and rounding.
• Multiplication pipeline: Usually three to four stages for partial product generation, reduction, and final addition and rounding.
• FMA pipeline: Often four to six stages combining the multiplication and addition flows.

Modern high-performance processors sustain one or more FMA operations per cycle through pipelining, multiple execution units, and SIMD. A core with two 512-bit FMA units, as in many of Intel's Xeon server processors, issues two eight-lane binary64 FMAs per cycle, which is 32 double-precision floating-point operations per cycle.

Addition Algorithm

Floating-point addition involves several steps:

1. Exponent comparison: Determine which operand has the larger exponent.
2. Alignment: Shift the smaller operand's significand right to align binary points.
3. Significand addition: Add or subtract the aligned significands based on signs.
4. Normalization: Shift the result to restore the leading one and adjust the exponent.
5. Rounding: Round the result to the destination precision.

Either the alignment shift or the normalization shift can be long, but never both in one operation: a large exponent difference leaves at most a one-place normalization, and massive cancellation occurs only when the exponents differ by at most one. High-performance adders exploit this with a far path and a close path computed in parallel.

Multiplication Algorithm

Floating-point multiplication is simpler than addition in some respects:

1. Exponent addition: Add the exponents and subtract the bias.
2. Significand multiplication: Multiply the significands using integer multiplication techniques.
3. Normalization: The product of two significands, each at least 1 and less than 2, is at least 1 and less than 4; when it is 2 or more, a one-place right shift with a matching exponent increment renormalizes it.
4. Rounding: Round the result, potentially requiring renormalization.
5. Sign computation: XOR the operand signs.

Division Algorithms

Floating-point division typically uses iterative algorithms:

• SRT division: A digit-recurrence method, usually radix 4 or higher in current FPUs, that retires two or more quotient bits per step, choosing each quotient digit from a small table indexed by leading bits of the partial remainder and the divisor.
• Newton-Raphson: Refines a reciprocal estimate with FMAs, converging quadratically, then multiplies by the dividend; a final FMA residual step yields the correctly rounded quotient.
• Goldschmidt: Converges quadratically, like Newton-Raphson, but its two multiplications per step are independent and can run in parallel or overlap in a pipelined multiplier, which lowers latency. It is not self-correcting, so rounding errors accumulate and designs carry extra guard bits.

Square root uses the same families of methods: a digit recurrence closely related to SRT division, or Newton-Raphson iteration on the reciprocal square root.

Precision Considerations

The number of significand bits determines the precision:

• Relative error bound: Under round to nearest, a single rounding of a result in the normal range has relative error at most the unit roundoff, u = 2^−p, where p is the full precision including the implicit leading bit. That is 2⁻²⁴ ≈ 5.96 × 10⁻⁸ for binary32 and 2⁻⁵³ ≈ 1.11 × 10⁻¹⁶ for binary64.
• Machine epsilon: The gap between 1 and the next larger number is 2^1−p, twice the unit roundoff. This is the value that C's FLT_EPSILON (about 1.19 × 10⁻⁷) and DBL_EPSILON (about 2.22 × 10⁻¹⁶) report, so check which convention a quoted error bound uses.
• Decimal digits: A precision of p bits corresponds to about p × log₁₀(2) ≈ 0.301 × p decimal digits. Converting decimal to binary and back preserves 6 significant digits in binary32 and 15 in binary64, while printing 9 and 17 significant digits respectively guarantees that a binary value converts back to itself exactly.
• Ulp size: The spacing between adjacent representable values, one unit in the last place, is constant within each binade, the interval from one power of two to the next, and doubles from each binade to the next. Relative spacing therefore varies by a factor of two across a binade.

Range Considerations

The exponent width determines the representable range:

• Dynamic range: The ratio of the largest to the smallest positive normal number, about 10⁹ for binary16, 10⁷⁶ for binary32, and 10⁶¹⁶ for binary64. Each added exponent bit roughly doubles the exponent range and so squares the dynamic range.
• Overflow threshold: The largest finite number. Results beyond it round to infinity or to that number, depending on the rounding attribute.
• Underflow threshold: The smallest positive normal number, below which results become subnormal and lose precision.

Format Selection Guidelines

Different applications favor different formats:

• Half precision (binary16): Suitable for graphics, image storage, and machine learning inference, where memory bandwidth is critical and precision requirements are modest. Its largest finite value of 65,504 makes overflow easy to reach, so computations in it need careful scaling.
• Bfloat16 and 8-bit formats: Not IEEE 754 formats, but widely used in machine learning. Bfloat16, developed at Google for its TPUs, keeps binary32's 8-bit exponent and therefore its range, with only 8 bits of precision, about 2.4 decimal digits. Eight-bit formats with 4- or 5-bit exponents, known as E4M3 and E5M2, push further on recent accelerators.
• Single precision (binary32): The workhorse for graphics, gaming, and many scientific applications. Offers good precision with efficient hardware support.
• Double precision (binary64): Standard for scientific and engineering computing and for applications requiring high accuracy. General-purpose CPUs execute it at full speed, with half as many SIMD lanes as binary32, while most consumer GPUs run it at a small fraction of their binary32 rate.
• Quadruple precision (binary128): Used for high-precision scientific computing, long-running simulations, and as an accumulator format. Usually implemented in software, although IBM's POWER9 and later processors execute binary128 arithmetic in hardware.

Mixed Precision Computing

Modern applications increasingly use mixed precision strategies:

• Compute in low precision: Perform bulk operations in half or single precision for speed.
• Accumulate in high precision: Use higher precision accumulators to maintain accuracy.
• Iterative refinement: Factor a matrix in low precision, then compute residuals in higher precision and correct the solution, recovering full accuracy for well-conditioned problems.
• Tensor cores: Matrix units that multiply in low precision and accumulate in higher precision. NVIDIA introduced them with its Volta GPUs in 2017, multiplying binary16 values and accumulating in binary32.

Why Decimal?

Binary floating-point cannot exactly represent many common decimal fractions:

• 0.1 in binary: The decimal value 0.1 is a repeating fraction in binary, so binary64 stores the nearest representable value, 0.1000000000000000055511151231257827 to 34 significant digits, and 0.1 + 0.2 evaluated in binary64 gives 0.30000000000000004 rather than 0.3.
• Accumulated errors: Financial calculations involving currency can accumulate significant errors when using binary floating-point.
• Prescribed rounding: Tax, accounting, and currency-conversion rules often prescribe rounding to a stated number of decimal places with a stated rule for ties, which decimal arithmetic reproduces directly.
• Human expectation: Users expect calculations with decimal values to produce exact decimal results.

IEEE 754 Decimal Formats

IEEE 754-2008 defines three decimal floating-point formats. Decimal64 and decimal128 are basic formats, while decimal32 is an interchange format intended for storage. The exponent limits below apply when the significand is written with one digit before the decimal point:

• Decimal32: 7 significant digits with an exponent range of −95 to +96.
• Decimal64: 16 significant digits with an exponent range of −383 to +384.
• Decimal128: 34 significant digits with an exponent range of −6143 to +6144.

Encoding Schemes

Two encoding schemes are defined for decimal floating-point. Both represent the same set of values and differ only in bit layout:

• Binary Integer Decimal (BID): Stores the significand as a binary integer. Proposed by Intel, it is preferred for software implementations because it uses existing binary integer hardware efficiently.
• Densely Packed Decimal (DPD): Stores the significand using a compression scheme that packs three decimal digits into 10 bits. Proposed by IBM, it is more efficient for hardware implementations and for conversion to and from character strings.

Hardware Support

Decimal floating-point hardware is available in some processors:

• IBM POWER processors: Include a dedicated decimal floating-point unit, beginning with POWER6 in 2007.
• IBM z/Architecture: Provides comprehensive decimal floating-point support for mainframe applications.
• Software libraries: Most platforms rely on software implementations, such as Intel's Decimal Floating-Point Math Library, which uses BID, and IBM's decNumber, which uses DPD. They are slower than hardware but widely available.

Quantum and Cohorts

Unlike binary floating-point, decimal floating-point can represent one value in several ways, and so preserves trailing zeros as significant digits. The exponent of a particular representation sets its quantum, the value of one unit in its last digit, which records the precision of the value:

• 1.20 differs from 1.2: The two are equal in value but distinct in representation, since 1.20 has a quantum of 0.01 and 1.2 a quantum of 0.1.
• Cohort: The set of representations of one value forms its cohort.
• Preferred exponent: For each operation the standard specifies a preferred exponent for an exact result. Thus 1.20 + 1.3 gives 2.50, keeping the finer quantum, and 1.20 × 2.0 gives 2.400, adding the operands' exponents.

Interval Arithmetic

Interval arithmetic represents values as bounds rather than points:

• Guaranteed enclosure: The true value is guaranteed to lie within the computed interval.
• Directed rounding: Uses round toward negative infinity for lower bounds and round toward positive infinity for upper bounds.
• Error tracking: Automatically tracks and bounds accumulated errors through computations.
• Standardization: IEEE Std 1788-2015 standardizes interval arithmetic.

Arbitrary Precision

When fixed-precision formats are insufficient, arbitrary precision libraries provide precision limited only by memory and time:

• GNU MPFR: Built on the GMP library, it provides correctly rounded arbitrary precision floating-point arithmetic.
• Variable precision: Each variable carries its own precision, and each result is correctly rounded to the precision of its destination.
• Exact arithmetic: Some libraries provide exact rational arithmetic, eliminating rounding entirely.

Reproducibility

Achieving reproducible floating-point results across different platforms and optimizations is challenging:

• Compiler optimizations: Different optimization levels may reorder operations, changing results.
• Parallel reduction: The order of accumulation in parallel algorithms affects results.
• Hardware choices: Whether multiply-add pairs are contracted into FMAs, and whether intermediates are held in wider registers such as the x87's 80-bit registers, changes results.
• Transcendental functions: Library implementations of functions like sin() and log() may differ.
• Reproducibility clause: The 2008 revision of IEEE 754 added a dedicated clause on reproducible floating-point results, recommending how programming languages can guarantee identical results across conforming implementations; the 2019 revision retained and refined it.

Posit Number System

The posit format, proposed by John Gustafson in 2017, is an alternative to IEEE 754 floating-point:

• Tapered precision: Higher precision near 1.0, lower precision at extremes.
• No NaN or infinity: Uses a single value, NaR (Not a Real), for exceptional cases.
• Claimed advantages: Proponents claim better accuracy per bit in many applications. Independent analyses find the advantage confined to values near 1 in magnitude, and note that precision varying with magnitude complicates classical error analysis.
• Research status: Still primarily a research topic with limited hardware support.

Verification and Testing

Floating-point implementations require extensive verification:

• Test vectors: Comprehensive test suites cover normal cases, boundary cases, and special values.
• Random testing: Stochastic testing reveals edge cases missed by directed tests.
• Formal verification: The 1994 Pentium FDIV flaw, caused by missing entries in the quotient-selection table of a radix-4 SRT divider, led Intel to take a $475 million charge and pushed processor makers toward mechanical proof of their floating-point units. The division algorithm of AMD's K5 was proved correct with the ACL2 theorem prover.
• Conformance test programs: William Kahan's Paranoia program and John Hauser's TestFloat check an implementation's arithmetic against IEEE 754 behavior; TestFloat compares results with Hauser's SoftFloat software implementation.

Performance Optimization

High-performance floating-point implementations employ various optimizations:

• Parallel execution units: Multiple FPUs execute simultaneously.
• SIMD processing: Single instruction operates on multiple floating-point values.
• Speculative execution: Predicts branches to keep pipelines full.
• Out-of-order execution: Reorders independent operations for efficiency.

Power Management

Floating-point operations consume significant power:

• Clock gating: Disables unused portions of the FPU to save power.
• Voltage scaling: Reduces voltage for less demanding workloads.
• Format-specific optimization: Lower precision formats use less power.
• Operand isolation: Prevents unnecessary switching in inactive datapaths.