You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current logging processes of qbeast-spark is only covering small cases (such as file filtering in OTreeIndexlistFiles operation).
Logs are a very important part of the code, since they help users and developers to understand what is happening, facilitate bug-solving, and help with a better understanding of the code flow. Especially in a framework such as Spark, in which is easy to get lost between the APIs and Catalyst Optimizations.
My proposal is the following:
Add Logs for Writing Processes. Classes such as SparkDataWriter ,QbeastMetadataOperation and SparkOTreeManager should log information about, at least, the stage they are in.
Add Logs for Reading. Affecting QuerySpaceBuilder, QueryExecutor and OTreeIndex.
Add Logs on each PR / code modification. Especially if they contain sensitive processes or they touch some of the critical pieces.
Spark already has a trait for writing such messages, it's called Logging, and this is the description:
/** * Utility trait for classes that want to log data. Creates a SLF4J logger for the class and allows * logging messages at different levels using methods that only evaluate parameters lazily if the * log level is enabled.*/traitLogging
The text was updated successfully, but these errors were encountered:
The current logging processes of
qbeast-spark
is only covering small cases (such as file filtering inOTreeIndex
listFiles
operation).Logs are a very important part of the code, since they help users and developers to understand what is happening, facilitate bug-solving, and help with a better understanding of the code flow. Especially in a framework such as Spark, in which is easy to get lost between the APIs and Catalyst Optimizations.
My proposal is the following:
SparkDataWriter
,QbeastMetadataOperation
andSparkOTreeManager
should log information about, at least, the stage they are in.Spark already has a
trait
for writing such messages, it's calledLogging
, and this is the description:The text was updated successfully, but these errors were encountered: