Error on run logs page when run has not started

[mtuchi]

Describe the bug

History.OpenFn.Lightning.-.10.March.2023.mp4

To Reproduce Steps to reproduce the behavior:

1. Go to the 'History page'
2. Click on 'A timeout job to view the run logs'
3. See the error

Expected behavior
View the run logs

Screenshots If applicable, add screenshots to help explain your problem.

Additional context

• Job sample code

sql(
  state =>
    `  SELECT * FROM patient`,
);

fn(state => {
  console.log('Data fetched from source database ✅ ')
  const results = state.response.body.rows;
  return { ...state, results };
  
});

• State. Check lastpass for OFG Demo Postgres DB
• Adaptor postgresql

[stuartc]

This appears to a be a side effect of some other unexpected behaviour:

1. The Scrubber is failing on the startup of a job, causing the Runtime to crash and set finished_at during it's cleanup.
2. The ObanManager is getting a meta.error value that isn't a map, causing Oban to throw an error too.

We need to reword this story to address these two issues.

One consideration needs to addressed, how do we want to deal with 'system' failures - like failing to start the Runtime or Scrubber...

[error] Oban exception:
:function_clause
    (elixir 1.14.2) lib/string.ex:2017: String.length(false)
    (elixir 1.14.2) lib/enum.ex:3257: anonymous fn/3 in Enum.sort_by/3
    (elixir 1.14.2) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir 1.14.2) lib/enum.ex:3257: Enum.sort_by/3
    (lightning 0.4.5) lib/lightning/scrubber.ex:60: anonymous fn/1 in Lightning.Scrubber.start_link/1
    (elixir 1.14.2) lib/agent/server.ex:8: Agent.Server.init/1
    (stdlib 4.2) gen_server.erl:851: :gen_server.init_it/2
    (stdlib 4.2) gen_server.erl:814: :gen_server.init_it/6
    (stdlib 4.2) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

meta:
  %{
  args: %{"attempt_run_id" => "80ab13bc-9853-413f-bc2e-c7c6b7444825"},
  attempt: 1,
  conf: %Oban.Config{
    dispatch_cooldown: 100,
    engine: Oban.Engines.Basic,
    get_dynamic_repo: nil,
    log: false,
    name: Oban,
    node: "fn",
    notifier: Oban.Notifiers.Postgres,
    peer: Oban.Peers.Postgres,
    plugins: [
      {Oban.Plugins.Cron,
       [
         crontab: [
           {"* * * * *", Lightning.Jobs.Scheduler},
           {"* * * * *", ObanPruner},
           {"0 2 * * *", Lightning.Accounts,
            [args: %{"type" => "purge_deleted"}]},
           {"0 10 * * *", Lightning.DigestEmailWorker,
            [args: %{"type" => "daily_project_digest"}]},
           {"0 10 * * MON", Lightning.DigestEmailWorker,
            [args: %{"type" => "weekly_project_digest"}]},
           {"0 10 1 * *", Lightning.DigestEmailWorker,
            [args: %{"type" => "monthly_project_digest"}]}
         ]
       ]}
    ],
    prefix: "public",
    queues: [
      scheduler: [limit: 1],
      workflow_failures: [limit: 1],
      background: [limit: 1],
      runs: [limit: 4]
    ],
    repo: Lightning.Repo,
    shutdown_grace_period: 60000,
    stage_interval: 1000,
    testing: :disabled
  },
  id: 39045,
  job: %Oban.Job{
    __meta__: #Ecto.Schema.Metadata<:loaded, "public", "oban_jobs">,
    id: 39045,
    state: "executing",
    queue: "runs",
    worker: "Lightning.Pipeline",
    args: %{"attempt_run_id" => "80ab13bc-9853-413f-bc2e-c7c6b7444825"},
    meta: %{},
    tags: [],
    errors: [],
    attempt: 1,
    attempted_by: ["fn"],
    max_attempts: 1,
    priority: 1,
    attempted_at: ~U[2023-03-12 20:40:35.446580Z],
    cancelled_at: nil,
    completed_at: nil,
    discarded_at: nil,
    inserted_at: ~U[2023-03-12 20:40:35.328955Z],
    scheduled_at: ~U[2023-03-12 20:40:35.328955Z],
    conf: %Oban.Config{
      dispatch_cooldown: 100,
      engine: Oban.Engines.Basic,
      get_dynamic_repo: nil,
      log: false,
      name: Oban,
      node: "fn",
      notifier: Oban.Notifiers.Postgres,
      peer: Oban.Peers.Postgres,
      plugins: [
        {Oban.Plugins.Cron,
         [
           crontab: [
             {"* * * * *", Lightning.Jobs.Scheduler},
             {"* * * * *", ObanPruner},
             {"0 2 * * *", Lightning.Accounts,
              [args: %{"type" => "purge_deleted"}]},
             {"0 10 * * *", Lightning.DigestEmailWorker,
              [args: %{"type" => "daily_project_digest"}]},
             {"0 10 * * MON", Lightning.DigestEmailWorker,
              [args: %{"type" => "weekly_project_digest"}]},
             {"0 10 1 * *", Lightning.DigestEmailWorker,
              [args: %{"type" => "monthly_project_digest"}]}
           ]
         ]}
      ],
      prefix: "public",
      queues: [
        scheduler: [limit: 1],
        workflow_failures: [limit: 1],
        background: [limit: 1],
        runs: [limit: 4]
      ],
      repo: Lightning.Repo,
      shutdown_grace_period: 60000,
      stage_interval: 1000,
      testing: :disabled
    },
    conflict?: false,
    replace: nil,
    unique: nil,
    unsaved_error: %{
      kind: {:EXIT, #PID<0.1081.0>},
      reason: :function_clause,
      stacktrace: [
        {String, :length, [false], [file: 'lib/string.ex', line: 2017]},
        {Enum, :"-sort_by/3-fun-1-", 3, [file: 'lib/enum.ex', line: 3257]},
        {Enum, :"-reduce/3-lists^foldl/2-0-", 3,
         [file: 'lib/enum.ex', line: 2468]},
        {Enum, :sort_by, 3, [file: 'lib/enum.ex', line: 3257]},
        {Lightning.Scrubber, :"-start_link/1-fun-0-", 1,
         [file: 'lib/lightning/scrubber.ex', line: 60]},
        {Agent.Server, :init, 1, [file: 'lib/agent/server.ex', line: 8]},
        {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 851]},
        {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 814]},
        {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}
      ]
    }
  },
  kind: :exit,
  max_attempts: 1,
  prefix: "public",
  queue: "runs",
  reason: :function_clause,
  state: :discard,
  tags: [],
  worker: "Lightning.Pipeline"
}
(%{duration: 98361157, queue_time: 117625000})

[error] Handler "oban-errors" has failed and has been detached. Class=:error
Reason={:badmap, :function_clause}
Stacktrace=[
  {Map, :get, [:function_clause, :reason, nil], [file: 'lib/map.ex', line: 534]},
  {Lightning.ObanManager, :handle_event, 4,
   [file: 'lib/lightning/oban_manager.ex', line: 32]},
  {:telemetry, :"-execute/3-fun-0-", 4,
   [
     file: '/home/openfn/Workspace/Lightning/deps/telemetry/src/telemetry.erl',
     line: 160
   ]},
  {:lists, :foreach_1, 2, [file: 'lists.erl', line: 1442]},
  {Oban.Queue.Executor, :emit_event, 1,
   [file: 'lib/oban/queue/executor.ex', line: 263]},
  {Oban.Backoff, :with_retry, 2, [file: 'lib/oban/backoff.ex', line: 38]},
  {Task.Supervised, :invoke_mfa, 2, [file: 'lib/task/supervised.ex', line: 89]},
  {Task.Supervised, :reply, 4, [file: 'lib/task/supervised.ex', line: 34]},
  {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}
]

[amberrignell]

The scrubber needs to ignore boolean values.