chore: temporarily exclude default-large vocab file
This commit is contained in:
2 files changed
+5
No files matched your search
@@ -12,6 +12,8 @@ homepage = "https://github.com/swaits/bpe-tokenizer/"
|
|||||||
repository = "https://github.com/swaits/bpe-tokenizer/"
|
repository = "https://github.com/swaits/bpe-tokenizer/"
|
||||||
rust-version = "1.64"
|
rust-version = "1.64"
|
||||||
license-file = "LICENSE"
|
license-file = "LICENSE"
|
||||||
|
exclude = ["vocab/multi.wiki.bpe.vs1000000.vocab"]
|
||||||
|
# ^ default-large vocab is temporarily excluded until we can get a crates.io exception
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
thiserror = "1.0.64"
|
thiserror = "1.0.64"
|
||||||
|
|||||||
@@ -9,6 +9,9 @@ frequent token pairs.
|
|||||||
It supports **Unicode-aware** text segmentation for sentence and word splitting,
|
It supports **Unicode-aware** text segmentation for sentence and word splitting,
|
||||||
making it suitable for processing a variety of languages and scripts.
|
making it suitable for processing a variety of languages and scripts.
|
||||||
|
|
||||||
|
**NOTE:** The `default-large` feature is temporarily disabled due to crate size
|
||||||
|
limitations; presently awaiting an exception from <help@crate.io>.
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
- **Bring your own BPE token vocabularies**, or use ...
|
- **Bring your own BPE token vocabularies**, or use ...
|
||||||
|
|||||||
Reference in new issue
Block a user