Unmarshal giving parse error on valid JSON

[andeke07]

I am trying to use this package for a server which will take JSON input and fire off an alert to our monitoring system. I have a fairly simple JSON package:

{
    "alert_summary": "uh oh",
    "detailed_description": "i dont know what to do",
    "deviceName": "test",
    "eventInfo": {
        "eventType": "disconnect",
        "more": "stuff",
        "there": "may",
        "be": "more",
        "values": "here"
    },
    "options": {
        "staging": true
    }
}

Within the eventInfo object, I will have an indeterminate number of fields depending on how much info the alert will be providing (hence why I found this package, I am considering everything within "eventInfo" to be "extra metadata" to add to the alert but I won't necessarily know what that info will be when the event is submitted.

I have the following code:

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"net/http"

	"github.com/go-playground/validator/v10"
	"github.com/perimeterx/marshmallow"
	"myorg/utils"
)

// Expected formatting of an incoming JSON event
type IncomingMessage struct {
	AlertSummary        string        `json:"alert_summary" validate:"required"`
	DetailedDescription string        `json:"detailed_description" validate:"required"`
	DeviceName          string        `json:"deviceName" validate:"required"`
	EventMetaData       struct {
		EventType string `json:"eventType" validate:"required"`
	} `json:"eventInfo" validate:"required"`
	Options             struct {
		AffectedCi   string `json:"affected_ci"`
		AffectedArea string `json:"affected_area"`
		HelpURL      string `json:"help_url"`
	} `json:"options"`
}

func StartServer(port string) (*http.Server, error) {
	srv := &http.Server{Addr: ":" + port}
	marshmallow.EnableCache()
	http.HandleFunc("/api/v1/submitRequest", handleIncomingEvent)
	http.ListenAndServe(":"+port, nil)
	return srv, nil
}

// Validates the incoming JSON for the appropriate formatting.
// If all ok, passes it on to be processed.
func handleIncomingEvent(w http.ResponseWriter, req *http.Request) {
	var incoming IncomingMessage
	var validate = validator.New()

	body, readingErr := io.ReadAll(req.Body)
	if readingErr != nil {
		fmt.Println("error reading body")
                return
	}

	result, unmarshallingErr := marshmallow.Unmarshal(body, &incoming)
	if unmarshallingErr != nil {
		utils.Log.Warnf("Could not unmarshal incoming data from %s: %s", req.Host, unmarshallingErr.Error())
		message := fmt.Sprintf("error decoding request body: %s", unmarshallingErr)
		returnMessage("error", http.StatusBadRequest, message, w)
		return
	}

	validationErr := validate.Struct(incoming)
	if validationErr != nil {
		utils.Log.Warnf("Bad Request Recieved from %s: %s", req.Host, validationErr.Error())
		message := fmt.Sprintf("input not in expected format: %s: %s", validationErr, validationErr.Error())
		returnMessage("error", http.StatusBadRequest, message, w)
		return
	}

	// If we get here do some stuff!
	// ...
	returnMessage("ok", http.StatusOK, "well done", w)

}

func returnMessage(status string, responseCode int, message string, w http.ResponseWriter) {
	var response OutgoingMessage
	response.Status = status
	response.ResponseCode = responseCode
	response.Message = message
	jsonResponse, _ := json.Marshal(response)
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(responseCode)
	w.Write(jsonResponse)
}

I am submitting a payload to my server with Insomnia however Marshmallow is giving me an error: Could not unmarshal incoming data from localhost: parse error: syntax error near offset 156 of ': "stuff"...'

I'm not entirely sure why this is happening. I realise that "more": "stuff" is not part of the struct but I was under the impression that would just be ignored when writing it in to the struct, and these values would then be available in the resultant map that also gets produced.

Is this a bug, or am I formatting my JSON incorrectly and/or handling it incorrectly?

Thanks.

[tmm1]

Can you reproduce on play.golang.org with some payload?

Seems like the input may be corrupted but its hard to tell without a standalone example

[avivpxi]

@andeke07 you are 100% correct. Here's a playground link reproducing the issue.
I see two separate issues here:

• Parsing issue on nested unknown fields - this one is an easy fix and it's waiting for us to settle on a solution before we merge - fix parse error on nested unknown fields #14.
• Representation of nested unknown fields - currently Marshmallow supports capturing unknown fields only at the root level. Any nested unknown fields are omitted. This means that in your use case the additional data in eventInfo will not be captured. The reason for this is that we want the data produced in the resulting map to be consistent with the schema of the struct. In your case, for example, we favor having resultMap["eventInfo"] containing an object of type struct {EventType string} over a map[string]interface{}. This prevents us from being able to represent unknown fields.

I think it should be resolved, but I wonder what will be the best approach. Changing this behavior at this point will be a breaking change so the default behavior should remain the same. At least until we release a new major version.

@andeke07 @tmm1 how would you feel about adding an additional option to determine whether the resulting map should aim to be schema-consistent, or contain all data?

[avivpxi]

Actually, I have an even better solution but let's merge the fix for now and open a new issue to discuss over there 🙏

[avivpxi]

fixed in v1.1.1