Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introducing TF v2 wrapper, ModelSignatureManager, and easy import of TF v2 models #2870

Merged

Conversation

maziyarpanahi
Copy link
Member

@maziyarpanahi maziyarpanahi commented May 11, 2021

This PR introduces the following features:

  • Updates existing loadSavedModel function to import TensorFlow SavedModel to be compatible with TF v2
  • Updates existing read and write TF v1 graph (freeze) to be compatible with TF v2 (currently crashes for none existing targets in TF v2 graph)
  • Implements ModelSignature to support different model providers such as TF Hub, HuggingFace, etc. (if they are for raw embedding with last_hidden_state or pooler_output) for SavedModel in TF v2 (currently only supports TF v1 with hard-coded placeholders for inputs and outputs)

Annotators adapted to these changes to support TF v2 SavedModel:

  • BertEmbeddings
  • BertSentenceEmbeddings
  • AlbertEmbeddings
  • XlnetEmbeddings
  • DistilBertEmbeddings

NOTE: All the previous models exported from TF v1 or uploaded as pretrained are still compatible. This PR will extend the support to TF v2 SavedModel especially the use of loadSavedModel in Python/Scala to easily import models from TF Hub and HuggingFace for the listed annotators.

NOTE2: There are other branches using TF v2 models which rely on this branch. (it was the only way to have support for TF v2 SavedModel for development)

########
This PR introduces a new annotator for RoBertaEmbeddings based on RoBERTa models (for instance roberta-base).

Import the following models:

  • roberta-base
  • roberta-large
  • distilroberta-base

########
This PR introduces a new annotator for DistilBertEmbeddings based on DistilBERT models (for instance distilbert-base-cased).

Benchmarks against bert_base_cased have shown better speed and very close accuracy in NER (faster convergence).

Import the following models:

  • distilbert-base-cased
  • distilbert-base-uncased
  • distilbert-base-multilingual-cased

########

Other PRs

wolliq and others added 30 commits March 29, 2021 16:52
# Conflicts:
#	docs/licensed/api/com/johnsnowlabs/license/Keys$.html
#	docs/licensed/api/com/johnsnowlabs/license/LicenseValidationException.html
#	docs/licensed/api/com/johnsnowlabs/license/LicenseValidator$.html
#	docs/licensed/api/com/johnsnowlabs/license/Licensed.html
#	docs/licensed/api/com/johnsnowlabs/ml/search/JSerializableKDTree.html
#	docs/licensed/api/com/johnsnowlabs/ml/search/Recipe.html
#	docs/licensed/api/com/johnsnowlabs/ml/search/SerializableKDTree.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/AssertionBatch.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/AssertionDatasetEncoder.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/TensorflowAssertion.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/TensorflowBertRE.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/TensorflowGenericClassifier.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/ActionInfo.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/Beam.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/DatabaseDefinition.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/HypMeta.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/PreprocessedDB.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/SchemaJSON.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/Sem2SQL.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/Edge.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/Grammar.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/Graph.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/Node.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/Utils$.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/Utils.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/action/A.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/action/Action.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/action/C.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/action/Filter.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/action/N.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/action/Order.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/action/Root.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/action/Root1.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/action/Sel.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/action/Sup.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/action/T.html
#	docs/licensed/api/com/johnsnowlabs/ml/tensorflow/text2sql/rule/lf.html
#	docs/licensed/api/com/johnsnowlabs/nlp/FeaturesAssembler.html
#	docs/licensed/api/com/johnsnowlabs/nlp/HasFeaturesJsl.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/Chunk2Token.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/DrugNormalizer$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/DrugNormalizer.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/Datapoint.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/dl/AssertionDLApproach$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/dl/AssertionDLApproach.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/dl/AssertionDLModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/dl/AssertionDLModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/dl/ReadablePretrainedAssertionDL.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/dl/ReadsAssertionGraph.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/logreg/AssertionLogRegApproach$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/logreg/AssertionLogRegApproach.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/logreg/AssertionLogRegModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/logreg/AssertionLogRegModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/logreg/Datapoint.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/logreg/NegexDatasetReader.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/logreg/ReadablePretrainedAssertionLogReg.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/logreg/Windowing$VectorizedChunk.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/assertion/logreg/Windowing.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/chunker/AssertionFilterer$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/chunker/AssertionFilterer.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/chunker/ChunkFilterer$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/chunker/ChunkFilterer.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/classification/DocumentLogRegClassifierApproach.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/classification/DocumentLogRegClassifierModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/classification/DocumentLogRegClassifierModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/classification/ReadablePretrainedDocumentLogRegClassifierModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/context/ContextualParserApproach$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/context/ContextualParserApproach.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/context/ContextualParserModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/context/ContextualParserModel$MatchedToken.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/context/ContextualParserModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/context/Dictionary.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/context/EntityDefinition.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/datasets/AssertionAnnotationAndText.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/datasets/AssertionAnnotationWithLabel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/datasets/AssertionAnnotationWithLabel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/datasets/IndexedChunk.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/DeIdentification$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/DeIdentification.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/DeIdentificationModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/DeIdentificationModel$StringReplacement.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/DeIdentificationModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/DeIdentificationParams.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/DefaultRegex$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/Obfuscator$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/ObfuscatorAnnotatorApproach$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/ObfuscatorAnnotatorApproach.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/ObfuscatorAnnotatorModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/ObfuscatorParams$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/ObfuscatorParams.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/ReadablePretrainedDeId.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/SentenceMaxException.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/StructuredDeid.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/StructuredDeidentification.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/TextToDocumentColumns.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/faker/Age$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/faker/Company$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/faker/Contact$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/faker/Faker.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/faker/ID$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/faker/Location$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/faker/Person$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/faker/Profession$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/deid/faker/URL$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/NerDisambiguator$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/NerDisambiguator.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/NerDisambiguatorModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/NerDisambiguatorModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/SimpleDisambiguationPipeline.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/datatypes/ContextMappingsSchema$$Record.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/datatypes/ContextMappingsSchema$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/datatypes/DisambiguationDataTypes$$DisambiguatedSentence.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/datatypes/DisambiguationDataTypes$$Disambiguation.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/datatypes/DisambiguationDataTypes$$DisambiguationResult.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/datatypes/DisambiguationDataTypes$$Position.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/datatypes/DisambiguationDataTypes$$RawDisambiguationResult.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/datatypes/DisambiguationDataTypes$$Sentence.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/datatypes/DisambiguationDataTypes$$SentenceEmbeddingWithType.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/datatypes/DisambiguationDataTypes$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/datatypes/LinalgAliases$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/linalg/package$$ManifoldTopologyBreeze$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/AnnotationLogic.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/Knowledge.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/KvKnowledgeExtractor.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/Mappings.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/PredictionLogic$Prediction.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/PredictionLogic.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/embeddings/EmbeddingsExtractor.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/embeddings/PoolingLogic.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/embeddings/impl/BowPoolingExtractor$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/embeddings/impl/BowPoolingExtractor.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/embeddings/impl/PoolingLogicBase.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/embeddings/impl/SentenceEmbeddingsExtractor$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/embeddings/impl/SentenceEmbeddingsExtractor.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/logic/embeddings/impl/SwitchableEmbeddingsExtractor.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/util/CategoryToDomainMapping$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/util/DisambiguatorModelParams.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/util/SwitchableSentenceEmbeddings$$SentenceEmbeddingColumns.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/util/SwitchableSentenceEmbeddings$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/util/linalg/CosineMetrics.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/util/linalg/CosineMetricsBreezeImpl.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/util/linalg/Topology.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/util/read/Reader.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/disambiguation/util/read/RocksDbReader.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/generic_classifier/GenericClassifierApproach$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/generic_classifier/GenericClassifierApproach.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/generic_classifier/GenericClassifierDataEncoder.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/generic_classifier/GenericClassifierModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/generic_classifier/GenericClassifierModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/generic_classifier/ReadablePretrainedGenericClassifier.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/generic_classifier/ReadsGenericClassifierGraph.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/merge/ChunkMergeApproach.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/merge/ChunkMergeModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/merge/ChunkMergeModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/merge/ReadablePretrainedChunkMerge.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/ner/IOBTagger$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/ner/IOBTagger.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/ner/NamedEntityConfidence.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/ner/NerChunker$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/ner/NerChunker.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/ner/NerConverterInternal$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/ner/NerConverterInternal.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/ner/NerTaggedInternal$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/ner/NerTagsEncodingInternal$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/BertREConfig.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/DLRelationInstance.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/PosologyREModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/REDataEncoder.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/REFeatureGenerator$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/RENerChunksFilter$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/RENerChunksFilter.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/ReadRelationExtractionDLModelTensorflowModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/ReadablePretrainedRelationExtractionDLModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/RelationExtractionApproach.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/RelationExtractionDLModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/RelationExtractionDLModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/RelationExtractionModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/RelationExtractionModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/re/RelationInstance.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/BigChunkEntityResolverApproach.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/BigChunkEntityResolverModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/BigChunkEntityResolverModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/BigFoundData.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/ChunkEntityResolverApproach.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/ChunkEntityResolverModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/ChunkEntityResolverModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/ConfidenceFunction$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/DistanceFunction$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/DistanceResult.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/JDataReader.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/JTreeComponent.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/JTreeData.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/JTreeReader.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/JTreeWriter.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/PoolingStrategy$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/ReadablePretrainedBigChunkEntityResolver.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/ReadablePretrainedChunkEntityResolver.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/ReadablePretrainedSentenceEntityResolver.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/SentenceEntityResolverApproach.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/SentenceEntityResolverModel$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/SentenceEntityResolverModel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/TreeData.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/params/ResolverParams.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/params/SentenceResolverParams.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/util/StringFunctions.html
#	docs/licensed/api/com/johnsnowlabs/nlp/annotators/resolution/util/StructFeature_HadoopFix.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/BaseChunk.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/BaseToken.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/EvaluationMetrics$TpFnFp.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/EvaluationMetrics.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/EvaluationModes$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/HasBeginEnd.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/POSEvaluation.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/ner/NerCrfEvaluation.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/ner/NerDLEvaluation.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/ner/NerDLMetrics.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/resolution/EvalEntityResolver.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/spell/NorvigSpellEvaluation.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/spell/SymSpellEvaluation.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/util/GoldTokenizer.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/util/LoggingData.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/util/TagsMetrics$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/eval/utils$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/jsl/AnnotatorType$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/jsl/SparkNLP$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/jsl/functions$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/pretrained/InternalsPythonResourceDownloader$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/serialization/ResolverMapFeature.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/AnnotationDefinition.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/AnnotationToolJsonReader$.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/AnnotationToolJsonReader.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/AnnotationValue.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/AnnotationValueChoices.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/AnnotationValueLabel.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/CantemistReader.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/CodiEspReader.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/CompletionDefinition.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/NerAnnotationDefinition.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/RelAnnotationDefinition.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/TaskDataDefinition.html
#	docs/licensed/api/com/johnsnowlabs/nlp/training/TaskDefinition.html
#	docs/licensed/api/com/johnsnowlabs/storage/JSLDatabase$.html
#	docs/licensed/api/com/johnsnowlabs/util/distances/StringDistances$.html
#	docs/licensed/api/com/johnsnowlabs/util/external/wmd4j/Cosine$.html
#	docs/licensed/api/com/johnsnowlabs/util/external/wmd4j/Euclidean$.html
#	docs/licensed/api/com/johnsnowlabs/util/wmd4j/FrequencyVector.html
#	docs/licensed/api/com/johnsnowlabs/util/wmd4j/SparkNLPWordVectors.html
#	docs/licensed/api/com/johnsnowlabs/util/wmd4j/WordMovers.html
#	docs/licensed/api/com/johnsnowlabs/util/wmd4j/WordVectors.html
#	docs/licensed/api/com/johnsnowlabs/util/wmd4j/emd/EarthMovers.html
#	docs/licensed/api/index.html
#	docs/licensed/api/index.js
#	docs/licensed/api/lib/diagrams.css
#	docs/licensed/api/lib/diagrams.js
#	docs/licensed/api/lib/index.css
#	docs/licensed/api/lib/index.js
#	docs/licensed/api/lib/ref-index.css
#	docs/licensed/api/lib/scheduler.js
#	docs/licensed/api/lib/template.css
#	docs/licensed/api/lib/template.js
maziyarpanahi and others added 28 commits May 12, 2021 11:08
BPE, input IndexToken, Output Array[TokenPiece]
End Index changed, prepend space for token without one
- remove cache until the Scala collection ops actually faster than re-doing the calculations again
- change merges type, it will be the same from the beginning to avoid splitting and converting into Map on every token
- also disabling a MosesTokenizerTest test
…-zero-index

Added fix for ending zero index in single char initial token in positional mask algorithm
Introducing RoBertaEmbeddings annotator based on RoBERTa models
Introducing DistilBertEmbeddings annotator based on DistilBERT models
@maziyarpanahi maziyarpanahi merged commit 3223efb into 310-release-candidate May 18, 2021
@KshitizGIT KshitizGIT deleted the feature/saved-model-bundle-auto-wrapper branch March 2, 2023 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new-feature Introducing a new feature on-hold cannot be merged right away
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants