Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lexiconfree beam search #101

Open
wants to merge 17 commits into
base: lattice_traces
Choose a base branch
from
Open

Conversation

SimBe195
Copy link
Collaborator

@SimBe195 SimBe195 commented Feb 19, 2025

Simple time synchronous beam search algorithm based on the new SearchAlgorithmV2 interface. Does not use (proper) pronunciation lexicon, word-level LM or transition model. Performs special handling of blank if a blank index is set. Main purpose is open vocabulary search with CTC/Neural Transducer (or similar) models.

Supports global pruning by max beam-size and by score difference to the best hypothesis. Uses a LabelScorer to context initialization/extension and scoring.

The search requires a lexicon that represents the vocabulary. Each lemma is viewed as a token with its index in the lexicon corresponding to the associated output index of the LabelScorer.

Depends on #103 and #104.

@larissakl
Copy link
Contributor

Two general points:

  1. How should this search algorithm be used? Are you planning to put the enum SearchTypeV2, the related ParameterChoice and Module_::createSearchAlgorithm() to Module.hh/.cc later? Or do you have a different plan?
  2. Would it maybe be a good idea to factor out the struct TimeStatistic and its related code so that it can be reused in other search algorithms?

@SimBe195
Copy link
Collaborator Author

SimBe195 commented Feb 20, 2025

1. How should this search algorithm be used? Are you planning to put the `enum SearchTypeV2`, the related `ParameterChoice` and `Module_::createSearchAlgorithm()` to Module.hh/.cc later? Or do you have a different plan?

With this PR alone, the search algorithm is not usable yet. I will make PR's for an Flf node and python bindings separately. But I can include the createSearchAlgorithm() function already here.

2. Would it maybe be a good idea to factor out the `struct TimeStatistic` and its related code so that it can be reused in other search algorithms?

Yeah, probably. Maybe even into Core?

@SimBe195 SimBe195 changed the base branch from master to lattice_traces March 5, 2025 14:15
Comment on lines +80 to +81
static const Core::ParameterBool paramUseSentenceEnd;
static const Core::ParameterBool paramSentenceEndIndex;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two are currently not used. I don't know if you want keep them if at some point you introduce sentence-end handling or if you want to remove them for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants