staviq
1a159553f9
tokenizer : special token handling ( #3538 )
...
* Rewrite special token handling from #1931
* shorten param name, add st verification by type
* use offsets instead of copy by substr
* formatting, remove copying iterator on delete
* llama : normalize code-style
* swift fix
* print pfx/sfx if verb, main: split pfx input sfx
* dont add space when using special tokens
* minor : comment + spacing
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-10-17 18:11:01 +03:00
..
2023-09-28 17:41:44 -04:00
2023-10-11 21:25:33 +03:00
2023-10-11 21:25:33 +03:00
2023-10-17 18:11:01 +03:00
2023-09-28 22:42:38 +03:00
2023-09-20 12:06:08 -04:00
2023-09-15 15:38:27 -04:00
2023-10-11 22:35:46 +03:00
2023-09-28 22:42:38 +03:00
2023-09-28 21:40:11 +03:00
2023-10-13 12:23:10 +02:00
2023-09-15 15:38:27 -04:00
2023-09-27 12:18:07 -04:00
2023-10-11 22:35:46 +03:00
2023-10-06 16:16:38 +03:00
2023-09-28 17:41:44 -04:00
2023-10-16 23:58:00 +03:00
2023-10-17 18:11:01 +03:00
2023-10-02 12:51:49 +03:00
2023-08-21 23:07:43 +03:00
2023-10-11 22:35:46 +03:00
2023-09-28 22:42:38 +03:00
2023-09-28 17:41:44 -04:00
2023-09-28 22:42:38 +03:00
2023-10-11 22:35:46 +03:00
2023-10-12 09:51:53 +03:00
2023-09-28 22:42:38 +03:00
2023-10-11 22:35:46 +03:00
2023-10-02 14:58:46 -04:00
2023-07-06 19:17:50 +03:00
2023-03-29 20:21:09 +03:00
2023-05-03 20:58:11 +03:00
2023-10-03 21:04:01 +03:00
2023-06-15 21:05:53 +03:00
2023-08-30 09:29:32 +03:00
2023-10-12 18:23:18 +03:00
2023-04-13 16:03:39 +03:00
2023-08-23 17:29:09 +03:00
2023-07-21 13:53:27 +03:00
2023-07-21 13:53:27 +03:00
2023-08-08 14:44:48 +03:00
2023-08-30 09:50:55 +03:00
2023-09-27 19:25:12 +03:00
2023-07-21 11:13:18 +03:00
2023-08-23 17:29:09 +03:00
2023-08-23 17:29:09 +03:00