WIP: Prospective demo

[picaultj]

TODO

• bug: (tab 3 analysis): mismatch of topic id between title+summary and topic evolution+analysis
• Tab2 (model configuration) : parametrize LLM-based analysis (choice of criterion - stored in a config file but not yet used)
• Harmonize location of storage of config files (feeds, models) for a user
• Tab3 (analysis): info about weak signals, strong signals: add link to document sources (url) to better explore the results
• Tab3 (analysis): split LLM information in tables for better display
• Tab4 (report generation): generate a newsletter based on a choice of selected topics, add possibility to edit/download/send by mail / subscribe
• bug? choice of reference_timestamp in reference_new_data
• add a feature to replay the past in addition to live data (useful to regenerate models from historical data)
• bug random sur process_new_data avec certaines données sur la sauvegarde parquet des df weak, strong ArrowTypeError: ("Expected bytes, got a 'tuple' object", 'Conversion failed for column Documents with type object')
• Refactor LLM call for detailed analysis: format in json to avoid passing the html template for formatting

[picaultj]

    # FIXME/TODO
    #   - self.topic_models shall not be an attribute of BERTrend -- to much memory consumption after a few iterations
    #   - what we did so far :
    #       * create topic models for each period, store them in self.topic_models
    #       * merge the data after preprocessing of each model
    #   - instead modify the functions as follows
    #       * train_topic_models: do a combined operation
    #           no need to store anything else than the last topic model (at least temporarily)
    #           for each period
    #               combine the operations of training the new topic model and merging
    #               optionnally store (as BERTopic serialization, using the function "save_topic_model" the newly created model
    #               merge the new one with previous data
    #           that way: no need to store BERTopic models inside BERTrend instance (memory saving)
    #                       we can serialize the BERTrend objects simply as a .drill and restore it the same way
    #    - in the demo, modify the different states of BERTrend (timestamps) are checked: use the BERTrend objects
    #       (ex. keys of self.doc_groups)
    #       instead of looking into the disk for available topic models