Text | TensorFlow

Text processing tools for TensorFlow

TensorFlow provides two libraries for text and natural language processing: KerasNLP and TensorFlow Text. KerasNLP is a high-level natural language processing (NLP) library that includes modern transformer-based models as well as lower-level tokenization utilities. It's the recommended solution for most NLP use cases. Built on TensorFlow Text, KerasNLP abstracts low-level text processing operations into an API that's designed for ease of use. But if you prefer not to work with the Keras API, or you need access to the lower-level text processing ops, you can use TensorFlow Text directly.

KerasNLP

import keras_nlp
import tensorflow_datasets as tfds

imdb_train, imdb_test = tfds.load(
  "imdb_reviews",
  split=["train", "test"],
  as_supervised=True,
  batch_size=16,
)
# Load a BERT model.
classifier = keras_nlp.models