Training & Alignment

Federated Learning

Federated learning trains a shared model across many decentralized devices or organizations that keep their data local, sending only model updates to a central server that averages them. The raw data never leaves the client.

What Federated Learning Is

Federated learning is a machine learning setting in which many clients, such as mobile phones or separate organizations, collaboratively train a single shared model while their training data stays decentralized on each client. Instead of collecting raw data into one central dataset, the central server coordinates training by repeatedly distributing the current model, receiving locally computed updates, and combining them. The data itself never moves.

The motivation is partly practical and partly about privacy. Much of the most useful training data, including the text people type, the photos on their phones, and records held inside hospitals or banks, is sensitive and often legally restricted from being pooled. Federated learning lets that data contribute to a model without being copied off the device or out of the institution. It also reduces the bandwidth and storage cost of shipping large raw datasets to a data center.

The term and the foundational method were introduced by McMahan and colleagues at Google in 2017. Their paper framed the problem as learning from data distributed across a large number of clients whose data is unbalanced and not independent and identically distributed, and whose communication with the server is the main bottleneck rather than computation.

Clients keep their raw data local and share only model parameters or updates.
A central server orchestrates training rounds and aggregates client contributions.
Originally targeted at sensitive, decentralized data like mobile keyboard text.
Communication, not local compute, is usually the limiting resource.

How a Federated Round Works (FedAvg)

The original and still most common algorithm is Federated Averaging, abbreviated FedAvg, from McMahan et al. 2017, titled "Communication-Efficient Learning of Deep Networks from Decentralized Data". A single round proceeds in four steps. The server sends the current global model weights to a random subset of available clients. Each selected client trains the model on its own local data for one or more passes of stochastic gradient descent. Each client sends its updated weights back to the server. The server aggregates these by a weighted average to form the next global model.

The key idea that makes FedAvg communication-efficient is that each client performs multiple local update steps before reporting back, rather than sending a gradient after every single batch. This reduces the number of communication rounds needed to reach a target accuracy, often by one to two orders of magnitude in the paper's experiments. Three hyperparameters govern the trade-off: C, the fraction of clients sampled per round; E, the number of local epochs each client runs; and B, the local minibatch size.

The weighting in the aggregation step is proportional to how much data each client holds. A client with more local examples pulls the global average toward its update more strongly. This weighted form is what distinguishes FedAvg from a naive uniform mean and is important when client datasets vary widely in size, which they almost always do in practice.

w_{t+1} = Σ_k (n_k / n) · w_{t+1}^k

FedAvg aggregation: the new global weights w_{t+1} are the average of each client k's locally updated weights w_{t+1}^k, weighted by n_k, the number of examples on client k, divided by n, the total number of examples across participating clients.

Step 1: server broadcasts the global model to a sampled subset of clients.
Step 2: each client trains locally for E epochs on minibatches of size B.
Step 3: clients return updated weights, not their raw data.
Step 4: server computes a data-size-weighted average to update the global model.

Why Non-IID Data and Communication Make It Hard

Federated learning breaks several assumptions that ordinary distributed training relies on. In a data center, the system controls how data is shuffled and split across workers, so each worker sees roughly the same distribution. In the federated setting the data is generated by the clients themselves and is heterogeneous, meaning it is not independent and identically distributed. One person types about cooking, another about sports, and a hospital in one region sees a different patient mix than another. This non-IID property can slow convergence and pull local updates in conflicting directions.

Communication is the dominant cost rather than computation. Clients are often phones or IoT devices on metered or slow connections that are only intermittently available, so the system tries to minimize the number of rounds and the size of each update. Two further obstacles are stragglers, clients that are slow or drop out mid-round, and system heterogeneity, the fact that clients differ widely in hardware, battery, and network, so the server cannot assume every selected client will finish on time.

Practical deployments respond to these constraints with careful scheduling and partial participation. Google's Gboard work, for example, only runs training when a device is idle, charging, and on an unmetered network, and the server proceeds with whatever subset of clients reports back in time rather than blocking on the slowest device.

Non-IID data: each client's local distribution differs, which can hurt convergence.
Communication rounds are expensive, so clients do extra local work per round.
Stragglers and dropouts mean the server must aggregate partial participation.
System heterogeneity: clients vary in compute, battery, and connectivity.

Cross-Device vs Cross-Silo

Federated learning is usually divided into two settings that look quite different in practice, a distinction made explicit in the survey "Advances and Open Problems in Federated Learning" by Kairouz, McMahan, and many co-authors. In the cross-device setting the clients are a very large number, potentially millions, of unreliable end-user devices such as smartphones. Any single device participates only occasionally, may drop out at any time, and is essentially anonymous to the server, so the system samples a fresh subset each round.

In the cross-silo setting the clients are a small number, often two to a few hundred, of organizations such as hospitals, banks, or companies. These clients are reliable, almost always available, and can be addressed by identity, so they can participate in most or all rounds. Cross-silo federated learning is common where data cannot be pooled for legal or competitive reasons but the participating institutions are stable and trusted to some degree.

The two settings change which engineering problems dominate. Cross-device work is mostly about scale, intermittent availability, and on-device constraints, while cross-silo work tends to emphasize stronger privacy guarantees, auditability, and handling a fixed set of large, statistically distinct data silos.

Cross-device: millions of unreliable phones, each sampled occasionally.
Cross-silo: a few stable organizations that join most rounds.
Cross-device emphasizes scale and intermittent participation.
Cross-silo emphasizes legal data isolation and stronger privacy guarantees.

Privacy: Federated Learning Alone Is Not Enough

Keeping raw data on the device is a meaningful privacy improvement, but it is not the same as full privacy. The model updates a client sends are derived from its private data, and research has repeatedly shown that updates can leak information, sometimes enough to reconstruct training examples or infer whether a specific record was used. So federated learning by itself is a partial protection, not a guarantee.

For this reason production systems combine federated learning with additional techniques. Secure aggregation, introduced by Bonawitz et al. in 2017, is a cryptographic protocol that lets the server compute only the sum or average of client updates without seeing any individual update, and it tolerates clients dropping out mid-protocol. Differential privacy adds calibrated noise so that the trained model is provably insensitive to any single client's data, bounding what can be inferred about any one participant.

These tools address different threats and are often layered together. Secure aggregation hides each individual update from the server during a round, while differential privacy limits what the final released model reveals about any contributor. The practical lesson is that federated learning is one component of a privacy-preserving pipeline rather than a complete solution on its own.

python

# One round of Federated Averaging (FedAvg), illustrative pseudocode
import numpy as np

def client_update(weights, local_data, epochs, lr):
    w = [layer.copy() for layer in weights]
    for _ in range(epochs):
        for x, y in local_data.minibatches():
            grads = compute_gradients(w, x, y)
            w = [wi - lr * gi for wi, gi in zip(w, grads)]
    return w  # only weights are sent back, never local_data

def federated_round(global_weights, clients, lr, epochs):
    updates, sizes = [], []
    for c in sample_clients(clients):          # partial participation
        w_k = client_update(global_weights, c.data, epochs, lr)
        updates.append(w_k)
        sizes.append(c.num_examples)           # n_k
    n = sum(sizes)                             # total examples this round
    # weighted average: w = sum_k (n_k / n) * w_k
    new_weights = [
        sum((nk / n) * w_k[layer] for w_k, nk in zip(updates, sizes))
        for layer in range(len(global_weights))
    ]
    return new_weights

A single federated round: the server samples clients, each trains locally and returns only its weights, and the server forms a data-size-weighted average. The raw local_data is never transmitted.

Raw data stays local, but model updates can still leak sensitive information.
Secure aggregation lets the server see only the combined update, not individual ones.
Differential privacy bounds what the released model reveals about any one client.
Strong deployments layer federated learning with both techniques, not just one.

Real Deployments

Federated learning moved from research to large-scale production fastest in mobile keyboards. Google described training next-word and query-suggestion models for Gboard, the Google Keyboard on Android, using federated learning so that the text people type stays on their phones. The production study by Hard et al. 2018, "Federated Learning for Mobile Keyboard Prediction", trained a recurrent neural network language model on-device and reported that the federated model matched or improved on a server-trained baseline while keeping data local.

Apple has published on private federated learning for on-device personalization, combining it with differential privacy and reporting deployment to large user populations for tasks such as model personalization. Its case study "Private Federated Learning In Real World Application" describes training an application-selection model where user data remains on the device and only privacy-protected updates are shared.

These deployments share a common pattern. The shared model improves from real user behavior at scale, training is scheduled to avoid disrupting the device, and privacy techniques are layered on top of the basic federated protocol. They also illustrate the limits, since both companies pair federated learning with additional safeguards rather than treating local data storage as sufficient privacy on its own.

Google trained Gboard next-word prediction with federated learning on Android.
Apple deployed private federated learning combined with differential privacy.
Training is scheduled for idle, charging, unmetered devices to avoid disruption.
Both pair federated learning with extra privacy safeguards, not just local storage.

Key takeaways

Federated learning trains a shared model across decentralized clients while their raw data stays local; only model updates are exchanged.
FedAvg (McMahan et al. 2017) runs rounds of broadcast, local training, and a data-size-weighted average: w_{t+1} = Σ_k (n_k/n) w_{t+1}^k.
Clients do multiple local steps per round to cut communication cost, the dominant bottleneck rather than compute.
Core challenges are non-IID data, communication limits, stragglers, and system heterogeneity across clients.
Cross-device (many unreliable phones) and cross-silo (few stable organizations) are the two main settings.
Keeping data local is not full privacy: updates can leak, so production systems add secure aggregation and differential privacy.

Frequently asked questions

In data-center distributed training, the operator controls the data and splits it evenly across workers, so each worker sees a similar distribution. In federated learning the data is generated and held by the clients, so it is non-IID, unbalanced, and only intermittently available. The server never sees the raw data and must tolerate clients dropping out mid-round.

Not on its own. Your raw data stays on your device, which is a real improvement, but the model updates sent to the server are derived from that data and can leak information, sometimes enough to reconstruct examples. For strong guarantees, federated learning is combined with secure aggregation, which hides individual updates, and differential privacy, which bounds what the final model reveals about any one user.

Federated Averaging is the foundational federated learning method from McMahan et al. 2017. In each round the server sends the current model to a sample of clients, each client trains locally for several steps, and the server averages the returned weights, weighting each client by how many examples it holds. Doing multiple local steps per round is what makes it communication-efficient.

Cross-device involves a very large number of unreliable end-user devices, such as millions of smartphones, where any one device participates only occasionally. Cross-silo involves a small number of reliable organizations, such as hospitals or banks, that can join most rounds. The cross-device setting is dominated by scale and availability, while cross-silo emphasizes legal data isolation and stronger privacy guarantees.

The clearest large-scale example is mobile keyboards: Google trained Gboard's next-word prediction with federated learning so typed text stays on the phone, reported in Hard et al. 2018. Apple has deployed private federated learning combined with differential privacy for on-device personalization tasks. Both schedule training for idle, charging, unmetered devices.

Each client's local data reflects that client's own behavior, so distributions differ widely and the usual independent-and-identically-distributed assumption fails. This can slow convergence and pull local updates in conflicting directions, since one client's optimum may not match another's. Handling this heterogeneity is one of the central open problems in federated learning research.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free