 glm94
|
019a7f44a8
|
Cleaned up the tokenizer and setting the stage for more granular token types.
|
2022-01-21 21:17:37 +00:00 |
|
|
glm94
|
0636315166
|
Starting to get a basic structure for tokenizing. Currently TK_Number tokens are being generated successfully.
|
2022-01-21 19:35:32 +00:00 |
|
|
glm94
|
72f6ec12e0
|
Tokenizing now strips the double quotes from strings, leaving only the intended value.
|
2022-01-20 22:10:48 +00:00 |
|
|
glm94
|
b32e1bad7c
|
Fixed some bugs in the string tokenizer.
|
2022-01-20 21:21:18 +00:00 |
|
|
glm94
|
23d4bd8089
|
Included a check for commas in the whitespace splitter. Might rename that function but this is all just testing before refactoring so I won't rename things just yet.
|
2022-01-20 19:48:24 +00:00 |
|
|
glm94
|
99ebd82996
|
Experimenting with tokenizing strings. This commit just focuses on splitting strings up by whitespace.
|
2022-01-20 18:17:58 +00:00 |
|