ReferenceWidget does not handle searches with null/None (#1014)

ast does not convert "null" to python's None and the following
Traceback is thrown:

Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module bika.lims.browser.widgets.referencewidget, line 204, in __call__
  Module bika.lims.browser.client.ajax, line 31, in __call__
  Module bika.lims.adapters.referencewidgetvocabulary, line 44, in __call__
  Module ast, line 80, in literal_eval
  Module ast, line 63, in _convert
  Module ast, line 62, in <genexpr>
  Module ast, line 60, in _convert
  Module ast, line 79, in _convert
ValueError: malformed string

ast.literal_eval has been replaced by built-in json.loads, and to
overcome the issue about unicodes:
#443
a function that converts unicodes to strings recursively has been
added, as suggested in the same issue above

Author: xispa

CHANGES.rst

@@ -21,8 +21,9 @@ Changelog
 
 **Fixed**
 
+- #1014 ReferenceWidget does not handle searches with null/None
 - #1008 Previous results from same batch are always displayed in reports
-- #1012 ARs and Samples from other clients are listed when logged in as contact
+- #1013 ARs and Samples from other clients are listed when logged in as contact
 - #991 New client contacts do not have access to their own AR Templates
 - #996 Hide checkbox labels on category expansion
 - #990 Fix client analysisspecs view

bika/lims/adapters/referencewidgetvocabulary.py

@@ -5,7 +5,6 @@
 # Copyright 2018 by it's authors.
 # Some rights reserved. See LICENSE.rst, CONTRIBUTORS.rst.
 
-import ast
 import json
 
 from zope.interface import implements
@@ -35,11 +34,13 @@ def __call__(self, result=None, specification=None, **kwargs):
         catalog_name = _c(self.request.get('catalog_name', 'portal_catalog'))
         catalog = getToolByName(self.context, catalog_name)
 
-        # N.B. We don't use json.loads to avoid unicode conversion, which will
-        #      fail in the catalog search for some cases
+        # json.loads does unicode conversion, which will fail in the catalog
+        # search for some cases. So we need to convert the strings to utf8
         # see: https://github.com/senaite/bika.lims/issues/443
-        base_query = ast.literal_eval(self.request['base_query'])
-        search_query = ast.literal_eval(self.request.get('search_query', "{}"))
+        base_query = json.loads(self.request['base_query'])
+        search_query = json.loads(self.request.get('search_query', "{}"))
+        base_query = self.to_utf8(base_query)
+        search_query = self.to_utf8(search_query)
 
         # first with all queries
         contentFilter = dict((k, v) for k, v in base_query.items())
@@ -129,3 +130,27 @@ def __call__(self, result=None, specification=None, **kwargs):
                         brains = _brains
 
         return brains
+
+    def to_utf8(self, data):
+        """
+        Convert unicode values to strings even if they belong to lists or dicts.
+        :param data: an object.
+        :return: The object with all unicode values converted to string.
+        """
+        # if this is a unicode string, return its string representation
+        if isinstance(data, unicode):
+            return data.encode('utf-8')
+
+        # if this is a list of values, return list of string values
+        if isinstance(data, list):
+            return [self.to_utf8(item) for item in data]
+
+        # if this is a dictionary, return dictionary of string keys and values
+        if isinstance(data, dict):
+            return {
+                self.to_utf8(key): self.to_utf8(value)
+                for key, value in data.iteritems()
+            }
+            # if it's anything else, return it in its original form
+
+        return data