[SPARKNLP-1113] Adding Text Reader

[danilojsl]

Description

This pull request introduces a new feature that enables reading and parsing Text files (.txt) into a structured Spark DataFrame. Leveraging this functionality allows for efficient processing and analysis of text files, seamlessly integrating with Spark NLP for enhanced downstream natural language processing tasks.

Important: Please do not merge this PR until #14491 is merged.

Motivation and Context

• Structured Data Representation: By transforming txt files content into a well-defined DataFrame structure, we enable seamless integration with Spark’s powerful analytical and data processing capabilities.
• Scalability: Leveraging Spark’s distributed architecture, this feature supports the efficient processing of large volumes of txt files data, which is critical for big data projects.
• Simplified Data Manipulation: Working with a structured DataFrame simplifies data manipulation tasks, such as filtering, aggregating, and transforming txt files content, reducing complexity and improving productivity.
• Enhanced Context for LLM Tasks: By organizing txt files into structured formats, we can curate and provide more specific, context-rich content for large language model (LLM) prompts. This improves the quality of LLM-generated outputs by allowing for more targeted and relevant contextual information, which is essential for applications like NLU and content generation.

How Has This Been Tested?

• Local Tests
• Google Colab notebook
• Databricks notebook

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Code improvements with no or little impact
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING page.
• I have added tests to cover my changes.
• All new and existing tests passed.