7.3.3. Tokens, Whitespace, and Comments

[10] token = identifier | keyword | boolean | number | character | string | named-constant | glyph-identifier | ( | ) | ' | . | ` | , | ,@

[11] delimiter = whitespace | ( | ) | [Quot] | ;

[12] whitespace = space | record-start | record-end | tab | form-feed | separator-character

[13] comment = ; any-character-except-record-end*

[14] atmosphere = whitespace | comment

[15] intertoken-space = atmosphere*

Whitespace characters are spaces, record starts, record ends, and separator-characters. Whitespace is used for improved readability and, as necessary, to separate tokens from each other, a token being an indivisible lexical unit such as an identifier or number, but is otherwise insignificant. Whitespace may occur between any two tokens, but not within a token. Whitespace may also occur inside a string, where it is significant.

A semicolon (;) indicates the start of a comment. The comment continues to the end of the record on which the semicolon appears. Comments are invisible, but the record end is visible as whitespace. This prevents a comment from appearing in the middle of an identifier or number.

intertoken-space may occur on either side of any token, but not within a token.

Tokens which require implicit termination (identifiers, numbers, characters, dot, and #! constants) may be terminated by any delimiter, but not necessarily by anything else.