Skip to content

feat: create a model from a tokenizer - #401

Merged
stephantul merged 3 commits into
mainfrom
distill-from-tokenizer
Oct 17, 2026
Merged

stephantul merged 3 commits into
mainfrom
distill-from-tokenizer

Conversation

@stephantul

Copy link
Copy Markdown
Contributor

I found myself often creating random models from tokenizers, so I decided to add a function that just takes a tokenizer and then creates random vectors. This is useful, because the vocabulary creation/guards and checks around vocabulary were not available outside of distillation.

I added two functions:

  1. random_init
  2. init_from_tokenizer

The former takes a tokenizer name, the second takes an initialized tokenizer.

@greptile-apps

greptile-apps Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] Adds new functions to initialize models from tokenizers.

The PR appears safe to merge, with two non-blocking improvements worth making.

Reviews (1) · Last reviewed commit: "mypy" · Reviewed by Greptile

@codecov

codecov Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
model2vec/distill/__init__.py 100.00% <100.00%> (ø)
model2vec/distill/distillation.py 100.00% <100.00%> (+5.47%) ⬆️
model2vec/distill/inference.py 98.59% <100.00%> (+<0.01%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread model2vec/distill/distillation.py Outdated
Comment thread model2vec/distill/distillation.py Outdated
@stephantul
stephantul requested a review from Pringled October 16, 2026 14:51

@Pringled Pringled left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One smoll thing but LGTM! I think some of the new tests could become part of existing tests but fine either way

Comment thread model2vec/distill/distillation.py Outdated
@stephantul
stephantul merged commit e53b8f5 into main Oct 17, 2026
11 checks passed
@stephantul
stephantul deleted the distill-from-tokenizer branch October 17, 2026 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants