Previously, the MiniYaml.From* helpers such as FromStream would consume the entire input and then return a list of top-level nodes. Now, the input is processed using deferred execution and top-level nodes are yielded as they are resolved from the input.
The motivating use-case is MapCache, which currently manually buffers nodes before passing to MiniYaml.FromString in order to improve responsiveness when large payloads are processed. Now that MiniYaml.FromStream yields results back as they come in, we can switch to that without disadvantage.
The maintains the performance where map cache can update search results as each node comes in over the network rather than having to wait for the entire batch to be transferred. Now we can also remove the string buffer that captures each node, reducing memory pressure and simplifying the code.
Currently removal nodes in MiniYaml are only applied whilst resolving inherited nodes. When merging collections to override existing nodes the removals are not resolved, resulting in duplicate key errors if you remove a node to then add it back with different values.
Now, removal nodes are also resolved when merging, allowing nodes to be removed as overridden during merging just like they can be with inheritance.
This rule recommends use of primary constructors. Apply the suggestions on simple POCOs, adjusting some to readonly/structs/records at the same time. For more complex classes, the use of primary constructors is more distracting than helpful, so silence the rule. IDEs can still offer fixes for using primary constructors, but it will not show up as a build issue.
During the merge operation, it is quite common to be dealing with a node that has no child nodes. When there are no such nodes, we can return early from some functions to avoid allocating new collections that will not be used.
In the MergePartial operation, reuse a dictionary as scratch space when checking for conflicts. We introduce a IntoDictionaryWithConflictLog helper to allow this. This avoids allocating a new dictionary for the conflict log that gets thrown away at each check.
- Previously the Inherits syntax was only resolved when used for top-level nodes. Now it is also resolved for nested nodes as well.
- Previously the MiniYAML Merge feature supported the ability to remove nodes, but this only worked within the context of inherited nodes. Now, we allow node removal to work outside of the inheritance context.
- Rename the filename parameter to name and make it mandatory. Review all callers and ensure a useful string is provided as input, to ensure sufficient context is included for logging and debugging. This can be a filename, url, or any arbitrary text so include whatever context seems reasonable.
- When several MiniYamls are created that have similar content, provide a shared string pool. This allows strings that are common between all the yaml to be shared, reducing long term memory usage. We also change the pool from a dictionary to a set. Originally a Dictionary had to be used so we could call TryGetValue to get a reference to the pooled string. Now that more recent versions of dotnet provide a TryGetValue on HashSet, we can use a set directly without the memory wasted by having to store both keys and values in a dictionary.
An empty MiniYaml comment that was indented was previously not recognized, and instead parsed as a key named '#'. Now, indented comments are recognized as comments, which matches the behaviour for unindented lines.
When handling the Nodes collection in MiniYaml, individual nodes are located via one of two methods:
// Lookup a single key with linear search.
var node = yaml.Nodes.FirstOrDefault(n => n.Key == "SomeKey");
// Convert to dictionary, expecting many key lookups.
var dict = nodes.ToDictionary();
// Lookup a single key in the dictionary.
var node = dict["SomeKey"];
To simplify lookup of individual keys via linear search, provide helper methods NodeWithKeyOrDefault and NodeWithKey. These helpers do the equivalent of Single{OrDefault} searches. Whilst this requires checking the whole list, it provides a useful correctness check. Two duplicated keys in TS yaml are fixed as a result. We can also optimize the helpers to not use LINQ, avoiding allocation of the delegate to search for a key.
Adjust existing code to use either lnear searches or dictionary lookups based on whether it will be resolving many keys. Resolving few keys can be done with linear searches to avoid building a dictionary. Resolving many keys should be done with a dictionary to avoid quaradtic runtime from repeated linear searches.
This changeset is motivated by a simple concept - get rid of the MiniYaml.Clone and MiniYamlNode.Clone methods to avoid deep copying yaml trees during merging. MiniYaml becoming immutable allows the merge function to reuse existing yaml trees rather than cloning them, saving on memory and improving merge performance. On initial loading the YAML for all maps is processed, so this provides a small reduction in initial loading time.
The rest of the changeset is dealing with the change in the exposed API surface. Some With* helper methods are introduced to allow creating new YAML from existing YAML. Areas of code that generated small amounts of YAML are able to transition directly to the immutable model without too much ceremony. Some use cases are far less ergonomic even with these helper methods and so a MiniYamlBuilder is introduced to retain mutable creation functionality. This allows those areas to continue to use the old mutable structures. The main users are the update rules and linting capabilities.
- Seal the classes, and make SourceLocation a readonly struct.
- In ToDictionary, use TryAdd to avoid a try-catch.
- In Merge, use ToList to ensure sources is only enumerated once.
Moving the key duplication check allows a redundant check on top-level nodes to be avoided. Add tests to ensure key checks are functioning as expected.
# Example Scenario
MockString:
CollectionOfStrings:
StringA: A
MockString:
CollectionOfStrings:
StringB: B
MockString:
-CollectionOfStrings:
MockString:
CollectionOfStrings:
StringC: C
MockString:
CollectionOfStrings:
StringD: D
# Previous MergePartial result
# The CollectionOfStrings is merged into a single unit, so the C and D items are dragged upwards and jump ahead of the Removal
# When this is processed, the final result removes CollectionOfStrings entirely
MockString:
CollectionOfStrings:
StringA: A
StringB: B
StringC: C
StringD: D
-CollectionOfStrings:
# New MergePartial result
# When merging nodes, we no longer allow merges to jump an intervening removal node.
# This means we can have multiple of a certain key (CollectionOfStrings in this example) which was not the case previously.
# When this is processed, the final result includes C/D but not A/B.
MockString:
CollectionOfStrings:
StringA: A
StringB: B
-CollectionOfStrings:
CollectionOfStrings:
StringC: C
StringD: D
- Stream lines in as memory rather than needing to realise a string for each line, via a new method in StreamExts.
- Use span to avoid string allocations during parsing until we want to realise the node itself, in MiniYaml.FromLines.
- Change several callsites to use the streaming extension method rather than string method where possible.
- Clone method will use the node count to create the correct capacity.
- ResolveInherits will use the node count as the suggested initial capacity.
- FromStream will now stream lines, rather than reading the whole file into memory and then splitting into lines.
These config files often contain many repeated strings which result in different string references in memory. By using a pool, we can detect when the strings are equal and reuse an existing reference as strings are immutable.
The FromLines will now use a pool to de-duplicate strings for a single call. By allowing a pool to be provided as a parameter, we can reuse even more strings. The MapCache defines such a pool so that strings are reused across all maps in the cache for even more savings.
