Tags: #go #json #serialization
json.Unmarshal when the JSON data is already in memory (like a string or []byte) and you know the data is small or moderate in size.json.NewDecoder for large JSON files, streaming data (such as from network requests), or when you need to handle data incrementally.json.Unmarshal reads the entire JSON into memory and decodes it directly into a Go variable, where as json.NewDecoder reads JSON data incrementally, parsing one token at a time, which is more memory-efficient with large JSON payloads.
In summary, use json.Unmarshal for simplicity with smaller data, and json.NewDecoder for efficiency with large or streaming data.
json.Marshal: encode data structure as JSON.json.Unmarshal: decode JSON into data structure.[!NOTE] This is also sometimes referred to as ‘serialize’ and ‘deserialize`.
json.Decoderjson.Decoder is for JSON streams.json.Decoder silently ignores invalid syntax.json.Decoder does not drain HTTP connections properly.[!NOTE] The issues with
json.Decoderare summarized from https://ahmet.im/blog/golang-json-decoder-pitfalls/
json.Decoder is for JSON streams (which are just concatenated/or new-line separated JSON values).
Example of a JSON stream:
{"Name": "Ed"}{"Name": "Sam"}{"Name": "Bob"}
Note: the entire content of that stream is not valid JSON (it should be inside a
[ ]to be a valid JSON value), BUT it is a valid JSON stream!
Lots of people have reported unexpected things happening because of how Decoder just silently ignores malformed JSON syntax. But I’ve not had an issue because I don’t really use Decoder so I can’t give a good example of how things can go wrong.
A poor example (which isn’t the same thing actually as silently ignoring malformed JSON syntax) would be that you expect each object in the stream to have an int field, but if it’s missing then to omit the field from the data structure. With Decoder it will utilize the zero value instead of just dropping the field altogether like you can with Unmarshal when using struct tags…
// Field is ignored by this package.
Field int `json:"-"`
// Field appears in JSON as key "myName".
Field int `json:"myName"`
// Field appears in JSON as key "myName" and
// the field is omitted from the object if its value is empty,
// as defined above.
Field int `json:"myName,omitempty"`
// Field appears in JSON as key "Field" (the default), but
// the field is skipped if empty.
// Note the leading comma.
Field int `json:",omitempty"`
This can slows down HTTP requests up to ~4x (although this is fixed by the time of Go 1.7).
If the HTTP endpoint is responding with a single JSON object and you are calling json.Decoder#Decode() only once (in which case you should be using json.Unmarshal() instead!), it means you are not getting io.EOF returned yet. Therefore you are not terminating json.Decoder by seeing that io.EOF and the response body remains open and therefore the TCP connection (or another Transport used) cannot be returned to the connection pool even though you are done with it.