ReferenceWidget does not handle searches with null/None

[xispa]

Description of the issue/feature this PR addresses

ast.literal_eval does not convert javascript's null to python's None and the following Traceback is thrown:

Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module bika.lims.browser.widgets.referencewidget, line 204, in __call__
  Module bika.lims.browser.client.ajax, line 31, in __call__
  Module bika.lims.adapters.referencewidgetvocabulary, line 44, in __call__
  Module ast, line 80, in literal_eval
  Module ast, line 63, in _convert
  Module ast, line 62, in <genexpr>
  Module ast, line 60, in _convert
  Module ast, line 79, in _convert
ValueError: malformed string

ast.literal_eval has been replaced by built-in json.loads, and to overcome the issue about unicodes, a function that converts unicodes to strings recursively has been added, as suggested in the same issue above.

Sometimes, searching catalog for a content type with a field that has either a value or None is highly desirable. E.g (in health):

client_uid = "01f004a407da40bebd9f63c2669abf9f"
query = dict(portal_type="Doctor", getPrimaryReferrerUID=[client_uid, None])
doctors = api.search(query, "portal_catalog")
results = map(lambda doctor: api.get_object(res).getPrimaryReferrerUID(), doctors)
list(set(results))
# This will return ["01f004a407da40bebd9f63c2669abf9f", None]

This search will include all Doctors that belong to the client client_uid and those that are not assigned to any client.

Current behavior before PR

ReferenceWidget does not supports searches with "null" values.

Desired behavior after PR is merged

ReferenceWidget does not supports searches with "null" values.

--
I confirm I have tested this PR thoroughly and coded it according to PEP8
and Plone's Python styleguide standards.

[ramonski]

Ok, just tested it here once again:

In [26]: data = {"name": 'S\xc3\x9cDRAD', "uids": ["1", "2", "3"], "foo": None}

In [27]: j = json.dumps(data)

In [28]: j
Out[28]: '{"foo": null, "name": "S\\u00dcDRAD", "uids": ["1", "2", "3"]}'

In [29]: ast.literal_eval(j)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-29-0491f871e43d> in <module>()
----> 1 ast.literal_eval(j)

/Users/rbartl/miniconda2/lib/python2.7/ast.pyc in literal_eval(node_or_string)
     78                 return left - right
     79         raise ValueError('malformed string')
---> 80     return _convert(node_or_string)
     81
     82

/Users/rbartl/miniconda2/lib/python2.7/ast.pyc in _convert(node)
     61         elif isinstance(node, Dict):
     62             return dict((_convert(k), _convert(v)) for k, v
---> 63                         in zip(node.keys, node.values))
     64         elif isinstance(node, Name):
     65             if node.id in _safe_names:

/Users/rbartl/miniconda2/lib/python2.7/ast.pyc in <genexpr>((k, v))
     60             return list(map(_convert, node.elts))
     61         elif isinstance(node, Dict):
---> 62             return dict((_convert(k), _convert(v)) for k, v
     63                         in zip(node.keys, node.values))
     64         elif isinstance(node, Name):

/Users/rbartl/miniconda2/lib/python2.7/ast.pyc in _convert(node)
     77             else:
     78                 return left - right
---> 79         raise ValueError('malformed string')
     80     return _convert(node_or_string)
     81

ValueError: malformed string

In [30]: json.loads(j)
Out[30]: {u'foo': None, u'name': u'S\xdcDRAD', u'uids': [u'1', u'2', u'3']}

So json.loads always returns unicode for strings and with the recursive encoding it works also for non-string values.

[ramonski]

Thanks!