Synthetic data.
Useful proxy. Not a free pass.

Artificially made does not mean one particular thing.

Synthetic data is data created rather than directly collected as the event or record it represents. It might come from a physics simulator, hand-written rules, statistical sampling, a generative model, or a transformation of existing examples. These sources behave differently, so the label alone says little about quality or privacy.

Before using any of them, name the intended job. Data good enough to exercise an interface may be unsuitable for estimating real-world accuracy, and a useful training example may not be safe to publish.

Use it to explore a specific gap.

Generated examples can help teams prototype a pipeline before sensitive records are available, exercise rare or hazardous scenarios in a simulator, or add variation when the real training set is narrow. In some workflows, models also create task examples that people can review before training.

The question is not whether the synthetic set looks like the original. It is whether the generated data helps with the target task, under the conditions that matter, without creating a new blind spot.

Repeatedly learning from generated outputs can narrow the next generation.

When a model trains on outputs from an earlier model, uncommon patterns can be lost or distorted over repeated rounds. Shumailov and colleagues demonstrated this model-collapse process in experiments with generative models. The result is a reason to track where training data came from—not proof that every synthetic dataset inevitably makes a model collapse.

A 2025 ICML study compared three workflows across Gaussian estimation, kernel-density estimation, and language-model fine-tuning. The results depended on whether each generation replaced the data that came before or kept it in the training set.

These findings show why “trained on synthetic data” is too broad a description. They reflect specific workflows and task settings; they do not predict every production pipeline or resolve separate questions about data quality, privacy, or provenance.

A practical safeguard is to preserve source data and lineage, control the share of generated examples, and evaluate each new model on an untouched test set that was not produced by the same generator.

Privacy needs its own evidence.

A generated record may not copy a real person line for line and still reveal patterns learned from sensitive training data. In a 2025 USENIX Security study, researchers tested fine-tuning on LLM-generated data using Pythia and OPT models, with email and law-domain data and specific extraction and membership-inference attacks. They found that some tested fine-tuning workflows increased measured leakage risk.

That result is specific to the study’s models, data, and threat setup. It is enough to reject a shortcut: calling a dataset “synthetic” does not establish that it is anonymous. If a privacy claim matters, state the attacker and access assumptions, test relevant attacks, and use formal privacy guarantees where appropriate.

Privacy and utility can pull in different directions. A dataset may reproduce useful fine detail because a model has learned it closely; that same detail can increase disclosure risk. Measure both for the actual release and use.

Make the data earn its place.

1. Write down the purpose. Separate prototyping, training, evaluation, and sharing; success in one does not establish suitability for another.
2. Record the source and method. Keep the original dataset or simulator version, generator and prompt, transformations, filters, and human review steps.
3. Check what is missing. Compare important groups, combinations, and rare cases. Similar averages can conceal a missing tail or a distorted relationship.
4. Test utility on real, held-out data. Compare with a real-data baseline and report performance by the cases that matter to the decision.
5. Test privacy separately. Define the threat model and evaluate extraction, membership, or reconstruction risks. Do not treat visual or statistical similarity as a privacy certificate.
6. Label and version every set. Keep generated and observed records identifiable in the pipeline so later teams can reproduce results and spot recursive feedback.

Synthetic data can make more experiments possible. It cannot supply evidence about a real population or replace a measurement that the research question depends on.

Follow data into the model and the evaluation.

Start with how machine learning fits patterns from examples ↗, then explore the separate risk of benchmark contamination ↗, the formal bounds offered by differential privacy ↗, and how to map privacy across an AI data path ↗.