chore: re-enable large default vocab feature

This commit is contained in:
swaits committed 2024-09-25 18:08:10 -06:00
1 parent 63a21e92a7
commit 8d742c90cb
2 files changed
+3 -9

No files matched your search

-3
View File
@@ -9,9 +9,6 @@ frequent token pairs.
It supports **Unicode-aware** text segmentation for sentence and word splitting,
making it suitable for processing a variety of languages and scripts.
**NOTE:** The `default-large` feature is temporarily disabled due to crate size
limitations; presently awaiting an exception from <help@crate.io>.
## Features
- **Bring your own BPE token vocabularies**, or use ...