Mapping python collections builtin methods to REST API patterns

[thorwhalen]

We have packages like dol and creek to help use build python-builtin-like interfaces to various data systems and formats.

That is, we aim at making operations on data should be done through a facade with maximum reuse between various use-case. Whenever possible (and not a ridiculous stretch) we chose our facade's base/inner_facing interface from python builtin interfaces. For example, from the abstract methods of python collections. from any data particulars.

Here, we want tools to map these python data facades to REST design patterns, automatically, via py2http tools.

Proposed interface:

def dispatch_collection(collection, resource_id) -> HttpServiceObj:
    "returns an http service object that has all the info needed to publish a web-service that enables users to interact with the collection"

In the above, we didn't have to specify any methods that need to be "exposed", because the function had it's own "routing" rules that associated, automatically, a set of methods for that collection (based, possibly, on it's type, name, and any other properties).
These rules should be parametrizable (plugin interface).

Optionally, the user can explicitly state the methods they want to expose.

In the example below, we want to be able to "list" objects as well as "get" them.
Perhaps the collection is a mapping, and we want to list keys (i.e. "ids", or "references") that can then be used to actually retrieve the object.

http_service_obj = dispatch_collection(collection, resource_id, methods=('__iter__', '__getitem__'))

[thorwhalen]

Dear Geppetto,

We have tools to map various data systems (local/remote files or blob storage, data bases, etc.) and formats to facades that use builtin python collections methods as their interface (as in, those listed in the "Collections Abstract Base Classes" table that can be found here: https://docs.python.org/3/library/collections.abc.html#collections-abstract-base-classes).

Now, we'd like to map these methods to standard, best-practice, REST web-service methods or design patterns.
Sometimes there could be an obvious 1-to-1 relationship between a python builtin method and a REST method, or sometimes the relationship might be many-to-1 (a sequence of python operations for one web service call) or 1-to-many (one python operation for a sequence of web service calls).

We'd like to have a list, as exhaustive as possible, that shows how we can map these python collection operations to, and from, REST web-service calls.

Geppetto:

c requirements of your application and the nature of your data.

1. Container.__contains__(item): This method checks if an item is in the container. In a RESTful web service, this could be mapped to a GET request that returns a boolean value. For example:

GET /container?item={item}

3. Container.__iter__(): This method returns an iterator over the items in the container. In a RESTful web service, this could be mapped to a GET request that returns a list of items. For example:

GET /container

4. Iterable.__iter__(): This method is similar to Container.iter(), but the context of an iterable rather than a container. The RESTful web service call would be the same:

GET /iterable

5. Collection.__len__(): This method returns the number of items in the collection. In a RESTful web service, this could be mapped to a GET request that returns an integer. For example:

GET /collection/length

6. Collection.__contains__(item): This method is similar to Container.contains(item), but it's used in the context of a collection rather than a container. The RESTful web service call would be the same:

GET /collection?item={item}

7. Callable.__call__(*args, **kwargs): This method calls the callable with the given arguments and keyword arguments. In a RESTful web service, this could be mapped to a POST request that includes the arguments and keyword arguments in the request body. For example:

POST /callable -d '{ "args": [...], "kwargs": {...} }'

9. Set.__contains__.(item): This method checks if an item is in the set. In a RESTful web service, this could be mapped to a GET request that returns a boolean value. For example:

GET /set?item={item}

11. Mapping.__getitem__(key): This method returns the value associated with a key in the mapping. In a RESTful web service, this could be mapped to a GET request that returns the value. For example:

GET /mapping/{key}

12. MutableMapping.__setitem__(key, value): This method sets the value associated with a key in the mutable mapping. In a RESTful web service, this could be mapped to a PUT or PATCH request that includes the value in the request body. For example:

PUT /mutablemapping/{key} -d `{ "value": ... } `

13. MutableMapping.__delitem__(key): This method deletes the item associated with a key in the mutable mapping. In a RESTful web service, this could be mapped to a DELETE request. For example:

DELETE /mutablemapping/{key}

14. Sequence.__getitem__(index): This method returns the item at a specific index in the sequence. In a RESTful web service, this could be mapped to a GET request that returns the item. For example:

GET /sequence/{index}

15. MutableSequence.__setitem__(index, value): This method sets the value of the item at a specific index in the mutable sequence. In a RESTful web service, this could be mapped to a PUTor PATCH request that includes the value in the request body. For example:

PUT /mutablesequence/{index} -d `{ "value": ... } `

16. MutableSequence.__delitem__(index): This method deletes the item at a specific index in the mutable sequence. In a RESTful web service, this could be mapped to a DELETE request. For example:

DELETE /mutablesequence/{index}

17. Mapping.__iter__(): This method returns an iterator over the keys in the mapping. In a RESTful web service, this could be mapped to a GET request that returns a list of keys. For example:

GET /mapping/keys

18. Mapping.__len__(): This method returns the number of items in the mapping. In a RESTful web service, this could be mapped to a GET request that returns an integer. For example:

GET /mapping/length

20. Mapping.__contains__(key): This method checks if a key is in the mapping. In a RESTful web service, this could be mapped to a GET request that returns a boolean value. For example:

GET /mapping/contains?key={key}

21. MutableSet.__contains__(item): This method checks if an item is in the mutable set. In a RESTful web service, this could be mapped to a GET request that returns a boolean value. For example:

GET /mutableset?item={item}

22. MutableSet.add(item): This method adds an item to the mutable set. In a RESTful web service, this could be mapped to a POST request that includes the item in the request body. For example:

POST /mutableset -d '{ "item": ... } '

23. MutableSet.remove(item): This method removes an item from the mutable set. In a RESTful web service, this could be mapped to a DELETE request. For example:

DELETE /mutableset?item={item}

24. Generator.__next__(): This method returns the next item from the generator. In a RESTful web service, this could be mapped to a GET request that returns the next item. For example:

GET /generator/next

[thorwhalen]

Our web-service is going to forward requests to a python backend, which is going to call one or several of these builtin collection methods. This may result in some errors, which need to then be forwarded to the the web-service caller, using various standard HTTP status codes, and standard message formats. Here again, we need to figure out what a good mapping from the python errors (type and message) to REST web-service (http status code and message) should be. What are the 10 most useful mappings we should implement?

Geppetto:

1. IndexError: This is raised when a sequence subscript is out of range.
   • HTTP Status Code: 404 Not Found
   • Message: "Resource not found."
2. TypeError: This is raised when an operation or function is applied to an object of inappropriate type.
   • HTTP Status Code: 400 Bad Request
   • Message: "Invalid type provided."
3. ValueError: This is raised when a built-in operation or function receives an argument that has the right type but an inappropriate value.
   • HTTP Status Code: 400 Bad Request
   • Message: "Invalid value provided."
4. KeyError: This is raised when a dictionary key is not found.
   • HTTP Status Code: 404 Not Found
   • Message: "Resource not found."
5. AttributeError: This is raised when an attribute reference or assignment fails.
   • HTTP Status Code: 400 Bad Request
   • Message: "Invalid attribute."
6. StopIteration: This is raised by built-in function next() and an iterator‘s next() method to signal that there are no further items produced by the iterator.
   • HTTP Status Code: 410 Gone
   • Message: "No more items to iterate."
7. NotImplementedError: This is raised when an abstract method that needs to be implemented in an inherited class is not actually implemented.
   • HTTP Status Code: 501 Not Implemented
   • Message: "Method not implemented."
8. MemoryError: This is raised when an operation runs out of memory.
   • HTTP Status Code: 507 Insufficient Storage
   • Message: "Server is out of memory."
9. TimeoutError: This is raised when a system function timed out at the system level.
   • HTTP Status Code: 408 Request Timeout
   • Message: "Request timed out."
10. PermissionError: This is raised when trying to open a file in write mode where write access is not allowed.
    • HTTP Status Code: 403 Forbidden
    • Message: "Permission denied."

[thorwhalen]

Here's some base python function that you can use to handle the error mapping described above.

dflt_error_mapping = {
    "IndexError": (404, "Resource not found."),
    "TypeError": (400, "Invalid type provided."),
    "ValueError": (400, "Invalid value provided."),
    "KeyError": (404, "Resource not found."),
    "AttributeError": (400, "Invalid attribute."),
    "StopIteration": (410, "No more items to iterate."),
    "NotImplementedError": (501, "Method not implemented."),
    "MemoryError": (507, "Server is out of memory."),
    "TimeoutError": (408, "Request timed out."),
    "PermissionError": (403, "Permission denied.")
}

def python_error_object_to_http_error_json(py_error: Exception, error_mapping: dict = None):
    """Takes a `py_error` object and returns some info for a http service error.
    That is, a json-serializable dict can then use in the http service response.
    

    This function does it's best to parse out extra information from the python error 
    message and use it as extra info in the web service response. 

    Example usage:

    >>> try:
    ...     # Some code that raises an exception...
    ...     raise ValueError("Invalid value: 'abc'")
    ... except Exception as e:
    ...     error_json = python_error_object_to_http_error_json(e)
    ...     print(json.dumps(error_json, indent=4))
    {
        "status": 400,
        "message": "Invalid value provided.",
        "details": "Invalid value: 'abc'"
    }
    """
    if error_mapping is None:
        error_mapping = dflt_error_mapping
        
    error_type = type(py_error).__name__
    if error_type in error_mapping:
        http_status_code, http_message = error_mapping[error_type]
        response = {
            "status": http_status_code,
            "message": http_message,
            "details": str(py_error)
        }
    else:
        # If the error type is not in the mapping, return a generic server error
        response = {
            "status": 500,
            "message": "Internal Server Error",
            "details": str(py_error)
        }
    return response

[thorwhalen]

Note that a mechanism to be able to dispatch any object has been added.

Here's an example:

handlers = [
    dict(
        endpoint=my_store,
        name='some_store',
        attr_names=['__iter__', '__getitem__'],
    ),
    my_func,
]
mk_app(handlers, **config)