Provenance, watermarking and fraud response

Lesson 9.2 ended with a detector that is right about one alert in thousands when it screens ordinary traffic, and that may miss content from a generator it has not seen. Organizations therefore need other ways to decide what to trust. This lesson covers two technical approaches that work from the other direction, by marking content at its source, and then the procedural controls, the payment and approval procedures that stop a convincing voice or face from moving money. It uses documented incidents and the guidance of financial and law enforcement agencies to show where those procedures fail and how to design them.

Content provenance with Content Credentials

The Coalition for Content Provenance and Authenticity (C2PA) publishes the technical specification for Content Credentials; this lesson refers to version 2.4, dated 1 April 2026 (Coalition for Content Provenance and Authenticity, 2026b). The specification describes provenance as a set of assertions, statements about an asset such as the device that captured it, the software that edited it, the actions performed and the earlier assets (ingredients) it was made from. The assertions are collected in a claim, which the claim generator signs on behalf of the signer with the signer's credential, an X.509 certificate. Assertions, claim and claim signature together form a C2PA Manifest, which is embedded in the file or stored externally in a manifest repository where a validator can locate it, for example through a URI that the claim generator adds to the file's XMP metadata. Figure 9.7 shows the structure.

The manifest is tied to the asset by a hard binding, a cryptographic hash over the asset's bytes or over specific parts of the file. Changing one bit of the protected content changes the hash, so a validator detects tampering after signing. A soft binding is computed from the content itself, an invisible watermark or a perceptual fingerprint, and lets a copy of the asset be matched to its manifest even when the bits differ, for example after re-encoding, or when the manifest has been stripped but a copy is kept in a provenance store. The specification states that a soft binding shall not be used as a hard binding, because it serves a different purpose. The basis of trust is the identity of the signer associated with the signing key, checked against lists of trusted certificate authorities and, with a trusted time stamp, against the validity period of the certificate.

What a valid manifest proves is narrower than it sounds. It shows that the assertions were signed by the holder of a particular credential and that the asset has not changed since. It does not show that the assertions are true, that the scene depicted happened, or that the signer acted in good faith; the specification states as a guiding principle that it should not provide value judgments about whether provenance data is good or bad, only about whether the assertions can be validated as associated with the asset, correctly formed and free from tampering. The C2PA explainer adds that provenance alone cannot tell whether content is true, accurate or factual, that adding provenance to an asset is optional, and that provenance metadata can be removed from an asset (Coalition for Content Provenance and Authenticity, 2026a). A missing manifest therefore proves nothing either, since authentic content need not carry one and any manifest can be stripped. NIST AI 100-4 (November 2024) notes that metadata is often stripped when files are shared on social media platforms, for deceptive or for benign reasons such as privacy (National Institute of Standards and Technology, 2024b).

Watermarking generated content

Provenance is attached by whoever signs the content. In the rest of this lesson, a watermark is a mark on generated content, embedded by the developer of a generator during or right after generation, so that the content can be recognized later even without metadata. It differs from the model watermark of Lesson 8.3, which marks a model to support an ownership claim. For text, Kirchenbauer et al. (2023) proposed a watermark for language models. Before each token is generated, a hash of the previous token seeds a pseudorandom split of the vocabulary into a green list, a fraction \(\gamma\) of all tokens, and a red list; the model's logits for green tokens are raised by a constant, so generated text contains more green tokens than chance would give. A detector that knows the hashing key, but not the model, counts the green tokens \(|s|_G\) in a text of \(T\) tokens and computes

which is approximately standard normal for human text, whose tokens fall on the green list at the chance rate \(\gamma\). With \(\gamma = 0.25\) and a 200-token text containing 90 green tokens, equation \eqref{eq:zscore} gives

far above the threshold \(z > 4\) that Kirchenbauer et al. use, at which the false positive rate is about \(3 \times 10^{-5}\). A human text of the same length would be expected to have about 50 green tokens. Figure 9.8 shows how the evidence grows with length.

The statistic also explains how the watermark can be removed. Every substituted token can break the green-list rule at its own position and, through the hash, at the next one, so in the analysis by Kirchenbauer et al. of a watermark with \(\gamma = 0.5\) that allows only green tokens, an attacker must modify roughly a quarter of the tokens of a long text to remove it. Generative attacks avoid that editing cost, for example asking the model to insert an emoji after every token and deleting the emojis afterward, which changes every hash. Kirchenbauer et al. describe such attacks as the strongest tools against watermarking known to them and suggest fine-tuning the model to refuse such requests. Paraphrasing removes the watermark from the passages that are rewritten, although Kirchenbauer et al. note that in a long text a few sentences left untouched can be enough for detection.

For images, watermarks can be added after generation or built into the generator. Stable Signature (Fernandez et al., 2023) fine-tunes the decoder of a latent diffusion model so that every image it produces carries a binary signature that a pretrained extractor recovers, and a statistical test decides whether an image came from that model. The authors report detecting the origin of a generated image cropped to 10% of its content with more than 90% accuracy at a false positive rate below \(10^{-6}\).

Removal is the weak point. Zhao et al. (2024) proposed regeneration attacks, which add noise to an image and reconstruct it with a denoiser or a generative model, and proved and showed empirically that pixel-level invisible watermarks are vulnerable to them while image quality is preserved. They suggest watermarks that keep the image only semantically similar as a possible defense. Saberi et al. (2024) proved a trade-off for watermarks that make only subtle changes to an image. Diffusion purification, adding Gaussian noise and denoising with a diffusion model, forces a detector to trade missed watermarks against false detections, and for watermarks with larger perturbations they built a model-substitution adversarial attack. They also showed spoofing, making real images be detected as watermarked, which could be used to discredit a developer. The WAVES benchmark (An et al., 2024) evaluated watermarks by the true positive rate at a false positive rate of 0.1%, together with image quality, and found, among other vulnerabilities, that watermarks relying on publicly available autoencoders could be removed with minimal image manipulation. NIST AI 100-4 reaches a similar conclusion, that most current watermarking schemes have been found vulnerable to removal. In its discussion of AI-generated child sexual abuse material and non-consensual intimate imagery, it adds that malicious actors often use freely available models, or build their own from open-source code, from which they can easily remove safeguards (National Institute of Standards and Technology, 2024b).

Watermarks are therefore useful evidence of origin when they are found, especially in cooperative settings such as a platform labeling its own model's output. They are not evidence of authenticity when they are absent, because an adversary can use a generator without a watermark or remove it.

Fraud with cloned voices and video calls

Fraud that uses synthetic media in real time to impersonate a trusted person is of direct concern to security teams. A widely reported case was described by CNN in February 2024, citing a briefing by Hong Kong police. A finance worker at a multinational firm took part in a video conference with what he believed were the company's chief financial officer and other members of staff, all of whom turned out to be deepfake recreations, and he agreed to remit a total of 200 million Hong Kong dollars, about 25.6 million US dollars (Chen and Magramo, 2024). In May 2024, the British engineering firm Arup confirmed to CNN that it was the company involved, that it had notified Hong Kong police in January, and that fake voices and images were used, and CNN reported, citing the public broadcaster RTHK and the police, that the money was sent in 15 transactions. According to the police, the worker had first suspected that an email purportedly from the company's UK-based chief financial officer was a phishing attempt, because it called for a secret transaction, but set his doubts aside after the call because the participants looked and sounded like colleagues he recognized; the fraud was discovered only when he later checked with the head office (Chen and Magramo, 2024; Magramo, 2024). Arup gave no further details, citing the ongoing investigation, and neither report describes how the video and voices were generated.

Agencies describe the same pattern more generally. The US Federal Bureau of Investigation (FBI) warned in December 2024 that criminals generate short audio clips of a relative's voice to ask for money in a crisis, use cloned voices to gain access to bank accounts, and generate video for real-time video chats with alleged company executives, law enforcement or other authority figures (Federal Bureau of Investigation, 2024). The US Financial Crimes Enforcement Network (FinCEN) reported in November 2024 that, beginning in 2023 and continuing in 2024, it had observed an increase in suspicious activity reports describing the suspected use of deepfake media, often in falsified identity documents used to circumvent identity verification. The same alert notes reports that criminals have used generative tools to impersonate an executive or other trusted employee and instruct victims to transfer large sums (Financial Crimes Enforcement Network, 2024).

Detection as one control among several

Detection has a place in this picture, but a modest one. FinCEN lists a photo or video flagged by deepfake detection software among its red flags, alongside indicators that do not depend on analyzing media at all, such as a customer who declines multifactor authentication, attempts to change communication methods during a live verification check because of excessive or suspicious technical glitches, or uses a third-party webcam plugin that can present previously generated video instead of live video (Financial Crimes Enforcement Network, 2024). It also notes that criminals may answer live verification prompts with tools that generate synthetic audio and video responses, although such responses may reveal inconsistencies. NIST AI 100-4 concludes that none of the content transparency techniques it reviews can be considered a comprehensive solution and that they must be applied in tandem with social efforts and initiatives (National Institute of Standards and Technology, 2024b). The reason detection plays only a modest part is the one from Lesson 9.2. A detector at a realistic base rate produces mostly false alarms, misses content from new generators, and can be evaded by an attacker who tests against it, while a payment that requires confirmation through an independent channel fails safe whatever the media looked like.

Response procedures

The controls that stop impersonation fraud verify the request, not the media. Figure 9.9 shows a procedure for payment and access requests; its elements are the following.

1. Call back on a known number. Verify any request to move money, change payment details or grant access by ending the conversation and calling the person back on a number from a trusted directory, never one supplied in the request. The FBI gives the same advice to individuals, to hang up, look up the organization's contact independently and call it directly (Federal Bureau of Investigation, 2024).
2. Confirm out of band. Require a second channel that the attacker is unlikely to control at the same time, such as a message through an authenticated internal system, an in-person confirmation, or a pre-agreed code phrase; the FBI suggests a secret word or phrase for families (Federal Bureau of Investigation, 2024).
3. Set approval thresholds. Above a set amount, or for any new beneficiary, require approval by a second authorized person who verifies independently, so that one convinced employee cannot complete the transfer alone.
4. Treat secrecy and urgency as red flags. A request to keep a transaction confidential or to bypass normal approval is itself a reason to escalate, as the "secret transaction" in the Hong Kong case should have been.
5. Never let one channel override another. A video or voice call that the requester arranged or initiated must not count as confirmation of a request that arrived by email, since the attacker may control both; in the Hong Kong case, the call was what dispelled the worker's doubts.
6. Report quickly, to the bank that sent the payment and to law enforcement; in the United States, the FBI asks victims to report to its Internet Crime Complaint Center with transaction details (Federal Bureau of Investigation, 2024).

These controls cost time and convenience, and an attacker can exploit the pressure to skip them, for example by claiming that a deal will collapse or by choosing a moment when the approver is traveling. The procedures therefore need management backing, so that an employee who delays a payment to verify it is protected, and periodic exercises in which the security team tests them with its own simulated requests.

What the lab shows

Lab 9 shows the fragility on which this lesson builds. A detector's signal comes from a trace of one generation process; a different generator leaves a different trace, and recompression can shift the scores enough to break a fixed threshold, as it did for the spectral detector trained on nearest-neighbor fakes. Watermarks are deliberate traces, and regeneration and purification can erase them in a similar way, which is why the response to impersonation fraud rests on procedures that do not need to know whether the media is synthetic.

Summary

Content Credentials bind signed assertions about origin and edits to an asset and reveal tampering after signing, but a valid manifest proves only who signed what, and a missing one proves nothing. Watermarks embedded by a generator, such as green-list watermarks for text and decoder-based watermarks for images, allow later detection with very low false positive rates, but regeneration, purification, paraphrasing and generative attacks can remove them, and an adversary can use a generator that adds none. The frauds with cloned voices and fabricated video calls described here succeed by persuading a person to approve a payment, and the effective defenses act on that approval, through call-back on known numbers, out-of-band confirmation, approval thresholds and escalation of secret or urgent requests, with detection and provenance as supporting signals.