Bug Report: Hello World with different backends gives different results

[matthewfeickert]

Description

While making a notebook for a talk I noticed that our Hello World example gives different results for different backends.

Expected Behavior

When running

pdf = pyhf.simplemodels.hepdata_like(signal_data=[12.,11.], bkg_data=[50.,52.], bkg_uncerts=[3.,7.])
*_, CLs_obs,CLs_exp = pyhf.utils.runOnePoint(1.0, [51, 48] + pdf.config.auxdata, pdf) # Python3
print('Observed: {} Expected: {}'.format(CLs_obs, CLs_exp[2]))

for the NumPy, TensorFlow, and PyTorch backends the results should be consistent.

Actual Behavior

The 3 backends give different results, with the NumPy backend giving very different results. This may be due to its divide by zero encountered in log problem(?).

NumPy backend
/home/mcf/Code/GitHub/pyhf/pyhf/tensor/numpy_backend.py:89: RuntimeWarning: divide by zero encountered in log
  return np.log(tensor_in)
/home/mcf/Code/GitHub/pyhf/pyhf/tensor/numpy_backend.py:163: RuntimeWarning: divide by zero encountered in log
  return np.exp(n*np.log(lam)-lam-gammaln(n+1.))
Observed: [0.05290116] Expected: [0.06445521]

TensorFlow backend
Observed: [0.04967977] Expected: [0.06119907]

PyTorch backend
Observed: 0.049736496061086655 Expected: 0.061249494552612305

Steps to Reproduce

import pyhf
import pyhf.simplemodels
import pyhf.utils

print('NumPy backend')
pdf = pyhf.simplemodels.hepdata_like(signal_data=[12.,11.], bkg_data=[50.,52.], bkg_uncerts=[3.,7.])
*_, CLs_obs,CLs_exp = pyhf.utils.runOnePoint(1.0, [51., 48.] + pdf.config.auxdata, pdf)
print('Observed: {} Expected: {}'.format(CLs_obs, CLs_exp[2]))

print('\nTensorFlow backend')
import tensorflow as tf
sess = tf.Session()
pyhf.set_backend(pyhf.tensor.tensorflow_backend(session=sess))

*_, CLs_obs,CLs_exp = pyhf.utils.runOnePoint(1.0, [51., 48.] + pdf.config.auxdata, pdf)
print('Observed: {} Expected: {}'.format(sess.run(CLs_obs), sess.run(CLs_exp[2])))

print('\nPyTorch backend')
pyhf.set_backend(pyhf.tensor.pytorch_backend())
*_, CLs_obs,CLs_exp = pyhf.utils.runOnePoint(1.0, [51., 48.] + pdf.config.auxdata, pdf)
print('Observed: {} Expected: {}'.format(CLs_obs[0], CLs_exp[2]))

produces

NumPy backend
/home/mcf/Code/GitHub/pyhf/pyhf/tensor/numpy_backend.py:89: RuntimeWarning: divide by zero encountered in log
  return np.log(tensor_in)
/home/mcf/Code/GitHub/pyhf/pyhf/tensor/numpy_backend.py:163: RuntimeWarning: divide by zero encountered in log
  return np.exp(n*np.log(lam)-lam-gammaln(n+1.))
Observed: [0.05290116] Expected: [0.06445521]

TensorFlow backend
Observed: [0.04967977] Expected: [0.06119907]

PyTorch backend
Observed: 0.049736496061086655 Expected: 0.061249494552612305

Checklist

• Run git fetch to get the most up to date version of master (on a recent branch)
• Searched through existing Issues to confirm this is not a duplicate issue
• Filled out the Description, Expected Behavior, Actual Behavior, and Steps to Reproduce sections above or have edited/removed them in a way that fully describes the issue

[lukasheinrich]

The issue is most likely the approximation of the poisson (poisson_from_normal). In the tests we set up the numpy backend using that flag to make the results comparable.

[matthewfeickert]

@lukasheinrich Yeah, that definitely contributes (thanks, I always forget this). Things look to be within tolerance given what we allow in the tests

import pyhf
import pyhf.simplemodels
import pyhf.utils

print('NumPy backend')
pyhf.set_backend(pyhf.tensor.numpy_backend(poisson_from_normal=True))
pdf = pyhf.simplemodels.hepdata_like(signal_data=[12.,11.], bkg_data=[50.,52.], bkg_uncerts=[3.,7.])
*_, CLs_obs,CLs_exp = pyhf.utils.runOnePoint(1.0, [51., 48.] + pdf.config.auxdata, pdf)
print('Observed: {} Expected: {}'.format(CLs_obs, CLs_exp[2]))

print('\nTensorFlow backend')
import tensorflow as tf
sess = tf.Session()
pyhf.set_backend(pyhf.tensor.tensorflow_backend(session=sess))

*_, CLs_obs,CLs_exp = pyhf.utils.runOnePoint(1.0, [51., 48.] + pdf.config.auxdata, pdf)
print('Observed: {} Expected: {}'.format(sess.run(CLs_obs), sess.run(CLs_exp[2])))

print('\nPyTorch backend')
pyhf.set_backend(pyhf.tensor.pytorch_backend())
*_, CLs_obs,CLs_exp = pyhf.utils.runOnePoint(1.0, [51., 48.] + pdf.config.auxdata, pdf)
print('Observed: {} Expected: {}'.format(CLs_obs[0], CLs_exp[2]))

now produces

NumPy backend
/home/mcf/Code/GitHub/pyhf/pyhf/tensor/numpy_backend.py:89: RuntimeWarning: divide by zero encountered in log
  return np.log(tensor_in)
/home/mcf/miniconda3/envs/pyhf/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:1652: RuntimeWarning: divide by zero encountered in true_divide
  x = np.asarray((x - loc)/scale, dtype=dtyp)
Observed: [0.05027371] Expected: [0.06127514]

TensorFlow backend
Observed: [0.04967977] Expected: [0.06119907]

PyTorch backend
Observed: 0.049736496061086655 Expected: 0.061249494552612305

Is it possible for us to set the backend from the CLI? Sorry, I haven't been fully following all what you and @kratsg have done there.

[lukasheinrich]

@matthewfeickert I agree it's confusing, not sure if maybe we should add a warning with non-numpy backends explaining the issue.

Probably both torch and TF support a better approximation than the normal one out of the box, so that would be a good improvement (in numpy we use a gamma-function based one)

currently we cannot set backend options from the cli, maybe we should be able to.

[lukasheinrich]

@matthewfeickert definitely a good FAQ item :-p

[matthewfeickert]

Thanks @lukasheinrich (I apparently need a lot more ☕ today). I'll go ahead and close this Issue and open up some ones that can be acted upon.