Facebook AI’s typography deepfake can scan real-life photos and emulate the text style

Facebook AI has released TextStyleBrush, an AI research project that copies the style of text in a photograph, based on just a single word. This means that the user can edit and replace text in imagery, and the tool can replicate both handwritten and typographic compositions and bring them into real-world scenes.

Currently a research project, the tool could one day have the potential for personalised messages and captions, as well as translating text in images to different languages and even real-time translation of street signs using AR. It also hopes to open the dialogue around detecting misuse of this sort of technology, “such as deepfake text attacks – a critical, emerging challenge in the AI field,” as stated in the research paper by Facebook AI, written by postdoctoral researcher Praveen Krishnan and research scientist Tal Hassner.

TextStyleBrush works in a similar fashion to style brush tools in word processors, but focusing more on the text aesthetic found in images. It also uses a generator architecture based on the StyleGAN2 model, and because TextStyleBrush is generating photo-realistic text images, there are a couple of limitations. “First, StyleGAN2 is an unconditional model, meaning it generates images by sampling a random latent vector. We, however, need to control the output based on two separate sources: our desired text content and style,” states the research. “Second, we have an added challenge of the unique nature of stylised text images. Representing text styles involves a combination of global information (e.g. colour palette and spatial transformation), along with detailed, fine-scale information, like the minute variations of individual penmanship.” To combat this, the generator is conditioned to the TextStyleBrush content and style.

Another challenge is that, due to the nature of real-life photography, there are limitless variables of text styles available. To address this, TextStyleBrush uses a typeface classifier, text recogniser and “adversarial discriminator”, explains the research. “We first measure how well our generator captures the sale of input text by using a pertained typeface classification network. Separately, we use a pertained text recognition network to evaluate the content of a generated image to reflect how well the generator captured the target content. Taken together, this allows for effective self-supervision of our training.”

Head here to read the full paper, TextStyleBrush: Transfer of text aesthetics form a single example. The handwriting data set can be downloaded here, and contains real-world handwritten samples from different writers.