BASIC token

A BASIC token is a single-byte representation of a BASIC keyword: Whenever the user edits or creates a BASIC line, any keywords are replaced by their respective token, and conversely; when the user LISTs the BASIC program, the tokens are displayed as the keywords they represent, in "plain text". Tokens only ever exist in the program storage area in memory, they are not visible to the user. Because they are a single byte, the resulting tokenized code requires less storage than the original text format.

BASIC tokens are not to be confused with BASIC keyword abbreviations.

Tokens are distinguished from "plain" PETSCII characters by the fact that token codes are always greater than or equal to 128/$80; i.e. the most significant bit in the byte representing a token is always set. This allows the interpreter to easily distinguish between tokens and other text at runtime. Note that only the keywords are tokenized in MS BASICs, other items like variable names, numeric and string constants, line numbers and other items remain in their original text format. Tokenization takes place when a line is entered or modified; the user's text is initially placed in a temporary area in memory, a buffer, and a routine known as the "chunker" copies it into the program storage area, replacing keywords with tokens as it goes.

Commodore BASIC V2 has 68 keywords and 8 operators, which are assigned token codes in the range from 128/$80 thru 203/$CB. Since the code 255/$FF is reserved for the "pi" character, this leaves 51 unused tokens in the range from 204–254/$CC–FE: Because of the ample use of vectors in the BASIC system, third-party BASIC expansions may use these for additional BASIC commands.

Example

If you type NEW, and then enter the BASIC program line

10 PRINT "HELLO WORLD"

the BASIC program memory (starting from 2049/$0801) will contain:

Location	Data	Description
2049/$0801	21/$15 8/$08	Pointer to beginning of "next" BASIC line, in low-byte/high-byte order
2051/$0803	10/$0A 0/$00	BASIC line number "10", in low-byte/high-byte order
2053/$0805	153/$99	The token for the PRINT keyword
2054/$0806	32/$20 34/$22	SPACE and quote characters following PRINT
2056/$0808 ... 2066/$0812		PETSCII codes for the "hello world" text
2067/$0813	34/$22	Quote at end of PRINTed text.
2068/$0814	0/$00	Zero-byte marking the end of the BASIC line.
2069/$0815	0/$00 0/$00	Two zero-bytes in place of the pointer to next BASIC line indicates the end of the program.

This token system saves space in memory and helps in slightly speeding up execution time, both in the computer and when the program is saved to and retrieved from tape or disk.

Token-to-keyword conversion table

BASIC 2.0

The following table lists all the reserved keywords and symbols, according to their associated token code:

128/$80 END 129/$81 FOR 130/$82 NEXT 131/$83 DATA 132/$84 INPUT# 133/$85 INPUT 134/$86 DIM 135/$87 READ 136/$88 LET 137/$89 GOTO 138/$8A RUN 139/$8B IF 140/$8C RESTORE 141/$8D GOSUB 142/$8E RETURN 143/$8F REM 144/$90 STOP 145/$91 ON 146/$92 WAIT

147/$93 LOAD 148/$94 SAVE 149/$95 VERIFY 150/$96 DEF 151/$97 POKE 152/$98 PRINT# 153/$99 PRINT 154/$9A CONT 155/$9B LIST 156/$9C CLR 157/$9D CMD 158/$9E SYS 159/$9F OPEN 160/$A0 CLOSE 161/$A1 GET 162/$A2 NEW 163/$A3 TAB( 164/$A4 TO 165/$A5 FN

166/$A6 SPC( 167/$A7 THEN 168/$A8 NOT 169/$A9 STEP 170/$AA + (Addition) 171/$AB − (Subtraction) 172/$AC * (Multiplication) 173/$AD / (Division) 174/$AE ↑ (Power) 175/$AF AND 176/$B0 OR 177/$B1 > (greater-than operator) 178/$B2 = (equals operator) 179/$B3 < (less-than operator) 180/$B4 SGN 181/$B5 INT 182/$B6 ABS 183/$B7 USR 184/$B8 FRE

185/$B9 POS 186/$BA SQR 187/$BB RND 188/$BC LOG 189/$BD EXP 190/$BE COS 191/$BF SIN 192/$C0 TAN 193/$C1 ATN 194/$C2 PEEK 195/$C3 LEN 196/$C4 STR$ 197/$C5 VAL 198/$C6 ASC 199/$C7 CHR$ 200/$C8 LEFT$ 201/$C9 RIGHT$ 202/$CA MID$ 203/$CB GO

Notice that the tokens break down into four "groups", which mirrors four of the "classes" of keywords/symbols.

• Commands are in the range 128–162/$80–A2
• Various "bywords" that form part of the syntax of the keywords above fall in the 163–169/$A3–$A9 range
• Arithmetic and logic operators have token codes 170–179/$AA–$B3
• Functions are in the range 180–202/$B4–CA

In BASIC ROM, the System variables TIME, TIME$, and STATUS, are handled as exceptions in the routines for handling "normal" variables.

GO (token code 203/$CB) breaks this "rule"; it is actually handled as a command, despite it's location "behind" the functions in the token system.

Other Commodore BASICs

Other Commodore BASICs define additional tokens. BASIC 3.5, found in the Plus/4 and Commodore 16, and BASIC 7.0, found in the Commodore 128, fill most of the available space. These two BASICs recognize many of the same keywords. BASIC 7.0 repurposes the 206 token; instead of RLUM, it uses 206 as a shift code to make several two-byte tokens. It also uses the 254 code, unused in BASIC 3.5, to make more two-byte tokens.

BASIC 4.0, found in some PETs, provides some additional keywords, some of which occur also in BASIC 3.5 and 7.0.

Token code	BASIC 3.5	BASIC 7.0	BASIC 4.0
204/$CC	RGR	RGR	CONCAT
205/$CD	RCLR	RCLR	DOPEN
206/$CE	RLUM	shift	DCLOSE
206/$CE 2/$02		POT
206/$CE 3/$03		BUMP
206/$CE 4/$04		PEN
206/$CE 5/$05		RSPPOS
206/$CE 6/$06		RSPRITE
206/$CE 7/$07		RSPCOLOR
206/$CE 8/$08		XOR
206/$CE 9/$09		RWINDOW
206/$CE 10/$0A		POINTER
207/$CF	JOY	JOY	RECORD
208/$D0	RDOT	RDOT	HEADER
209/$D1	DEC	DEC	COLLECT
210/$D2	HEX$	HEX$	BACKUP
211/$D3	ERR$	ERR$	COPY
212/$D4	INSTR	INSTR	APPEND
213/$D5	ELSE	ELSE	DSAVE
214/$D6	RESUME	RESUME	DLOAD
215/$D7	TRAP	TRAP	CATALOG
216/$D8	TRON	TRON	RENAME
217/$D9	TROFF	TROFF	SCRATCH
218/$DA	SOUND	SOUND	DIRECTORY
219/$DB	VOL	VOL
220/$DC	AUTO	AUTO
221/$DD	PUDEF	PUDEF
222/$DE	GRAPHIC	GRAPHIC
223/$DF	PAINT	PAINT
224/$E0	CHAR	CHAR
225/$E1	BOX	BOX
226/$E2	CIRCLE	CIRCLE
227/$E3	GSHAPE	GSHAPE
228/$E4	SSHAPE	SSHAPE
229/$E5	DRAW	DRAW
230/$E6	LOCATE	LOCATE
231/$E7	COLOR	COLOR
232/$E8	SCNCLR	SCNCLR
233/$E9	SCALE	SCALE
234/$EA	HELP	HELP
235/$EB	DO	DO
236/$EC	LOOP	LOOP
237/$ED	EXIT	EXIT
238/$EE	DIRECTORY	DIRECTORY
239/$EF	DSAVE	DSAVE
240/$F0	DLOAD	DLOAD
241/$F1	HEADER	HEADER
242/$F2	SCRATCH	SCRATCH
243/$F3	COLLECT	COLLECT
244/$F4	COPY	COPY
245/$F5	RENAME	RENAME
246/$F6	BACKUP	BACKUP
247/$F7	DELETE	DELETE
248/$F8	RENUMBER	RENUMBER
249/$F9	KEY	KEY
250/$FA	MONITOR	MONITOR
251/$FB	USING	USING
252/$FC	UNTIL	UNTIL
253/$FD	WHILE	WHILE
254/$FE	none	shift
254/$FE 2/$02		BANK
254/$FE 3/$03		FILTER
254/$FE 4/$04		PLAY
254/$FE 5/$05		TEMPO
254/$FE 6/$06		MOVSPR
254/$FE 7/$07		SPRITE
254/$FE 8/$08		SPRCOLOR
254/$FE 9/$09		RREG
254/$FE 10/$0A		ENVELOPE
254/$FE 11/$0B		SLEEP
254/$FE 12/$0C		CATALOG
254/$FE 13/$0D		DOPEN
254/$FE 14/$0E		APPEND
254/$FE 15/$0F		DCLOSE
254/$FE 16/$10		BSAVE
254/$FE 17/$11		BLOAD
254/$FE 18/$12		RECORD
254/$FE 19/$13		CONCAT
254/$FE 20/$14		DVERIFY
254/$FE 21/$15		DCLEAR
254/$FE 22/$16		SPRSAV
254/$FE 23/$17		COLLISION
254/$FE 24/$18		BEGIN
254/$FE 25/$19		BEND
254/$FE 26/$1A		WINDOW
254/$FE 27/$1B		BOOT
254/$FE 28/$1C		WIDTH
254/$FE 29/$1D		SPRDEF
254/$FE 30/$1E		QUIT
254/$FE 31/$1F		STASH
254/$FE 32/$20		none
254/$FE 33/$21		FETCH
254/$FE 34/$22		none
254/$FE 35/$23		SWAP
254/$FE 36/$24		OFF
254/$FE 37/$25		FAST
254/$FE 38/$26		SLOW

All of the additional tokens in BASIC 4.0 occur in BASIC 7.0, but with different token codes. Some also occur in BASIC 3.5.

Keyword	BASIC 4.0	BASIC 3.5	BASIC 7.0
CONCAT	204/$CC		254/$FE 19/$13
DOPEN	205/$CD		254/$FE 13/$0D
DCLOSE	206/$CE		254/$FE 15/$0F
RECORD	207/$CF		254/$FE 18/$02
HEADER	208/$D0	241/$F1	241/$F1
COLLECT	209/$D1	243/$F3	243/$F3
BACKUP	210/$D2	246/$F6	246/$F6
COPY	211/$D3	244/$F4	244/$F4
APPEND	212/$D4		254/$FE 14/$0E
DSAVE	213/$D5	239/$EF	239/$EF
DLOAD	214/$D6	240/$F0	240/$F0
CATALOG	215/$D7		254/$FE 12/$0C
RENAME	216/$D8	245/$F5	245/$F5
SCRATCH	217/$D9	242/$F2	242/$F2
DIRECTORY	218/$DA	238/$EE	238/$EE