Raw text

Tokenization happens first. The controls above are applied afterwards. The browser version mirrors the spaCy splits used in the Python example below.

Tokens after your choices

Vocabulary
Tokens
-
Types
-
Type/token ratio
-
Sentences
-

Hover over a token to inspect is_stop, is_punct, lemma_, and its sentence number. New and York remain separate tokens. Deciding that together they name one place is a job for a later step, not for the tokenizer.