During the merge operation, it is quite common to be dealing with a node that has no child nodes. When there are no such nodes, we can return early from some functions to avoid allocating new collections that will not be used.
In the MergePartial operation, reuse a dictionary as scratch space when checking for conflicts. We introduce a IntoDictionaryWithConflictLog helper to allow this. This avoids allocating a new dictionary for the conflict log that gets thrown away at each check.
- Previously the Inherits syntax was only resolved when used for top-level nodes. Now it is also resolved for nested nodes as well.
- Previously the MiniYAML Merge feature supported the ability to remove nodes, but this only worked within the context of inherited nodes. Now, we allow node removal to work outside of the inheritance context.
- Rename the filename parameter to name and make it mandatory. Review all callers and ensure a useful string is provided as input, to ensure sufficient context is included for logging and debugging. This can be a filename, url, or any arbitrary text so include whatever context seems reasonable.
- When several MiniYamls are created that have similar content, provide a shared string pool. This allows strings that are common between all the yaml to be shared, reducing long term memory usage. We also change the pool from a dictionary to a set. Originally a Dictionary had to be used so we could call TryGetValue to get a reference to the pooled string. Now that more recent versions of dotnet provide a TryGetValue on HashSet, we can use a set directly without the memory wasted by having to store both keys and values in a dictionary.
An empty MiniYaml comment that was indented was previously not recognized, and instead parsed as a key named '#'. Now, indented comments are recognized as comments, which matches the behaviour for unindented lines.
When handling the Nodes collection in MiniYaml, individual nodes are located via one of two methods:
// Lookup a single key with linear search.
var node = yaml.Nodes.FirstOrDefault(n => n.Key == "SomeKey");
// Convert to dictionary, expecting many key lookups.
var dict = nodes.ToDictionary();
// Lookup a single key in the dictionary.
var node = dict["SomeKey"];
To simplify lookup of individual keys via linear search, provide helper methods NodeWithKeyOrDefault and NodeWithKey. These helpers do the equivalent of Single{OrDefault} searches. Whilst this requires checking the whole list, it provides a useful correctness check. Two duplicated keys in TS yaml are fixed as a result. We can also optimize the helpers to not use LINQ, avoiding allocation of the delegate to search for a key.
Adjust existing code to use either lnear searches or dictionary lookups based on whether it will be resolving many keys. Resolving few keys can be done with linear searches to avoid building a dictionary. Resolving many keys should be done with a dictionary to avoid quaradtic runtime from repeated linear searches.
This changeset is motivated by a simple concept - get rid of the MiniYaml.Clone and MiniYamlNode.Clone methods to avoid deep copying yaml trees during merging. MiniYaml becoming immutable allows the merge function to reuse existing yaml trees rather than cloning them, saving on memory and improving merge performance. On initial loading the YAML for all maps is processed, so this provides a small reduction in initial loading time.
The rest of the changeset is dealing with the change in the exposed API surface. Some With* helper methods are introduced to allow creating new YAML from existing YAML. Areas of code that generated small amounts of YAML are able to transition directly to the immutable model without too much ceremony. Some use cases are far less ergonomic even with these helper methods and so a MiniYamlBuilder is introduced to retain mutable creation functionality. This allows those areas to continue to use the old mutable structures. The main users are the update rules and linting capabilities.
- Seal the classes, and make SourceLocation a readonly struct.
- In ToDictionary, use TryAdd to avoid a try-catch.
- In Merge, use ToList to ensure sources is only enumerated once.
Moving the key duplication check allows a redundant check on top-level nodes to be avoided. Add tests to ensure key checks are functioning as expected.
# Example Scenario
MockString:
CollectionOfStrings:
StringA: A
MockString:
CollectionOfStrings:
StringB: B
MockString:
-CollectionOfStrings:
MockString:
CollectionOfStrings:
StringC: C
MockString:
CollectionOfStrings:
StringD: D
# Previous MergePartial result
# The CollectionOfStrings is merged into a single unit, so the C and D items are dragged upwards and jump ahead of the Removal
# When this is processed, the final result removes CollectionOfStrings entirely
MockString:
CollectionOfStrings:
StringA: A
StringB: B
StringC: C
StringD: D
-CollectionOfStrings:
# New MergePartial result
# When merging nodes, we no longer allow merges to jump an intervening removal node.
# This means we can have multiple of a certain key (CollectionOfStrings in this example) which was not the case previously.
# When this is processed, the final result includes C/D but not A/B.
MockString:
CollectionOfStrings:
StringA: A
StringB: B
-CollectionOfStrings:
CollectionOfStrings:
StringC: C
StringD: D
- Stream lines in as memory rather than needing to realise a string for each line, via a new method in StreamExts.
- Use span to avoid string allocations during parsing until we want to realise the node itself, in MiniYaml.FromLines.
- Change several callsites to use the streaming extension method rather than string method where possible.
- Clone method will use the node count to create the correct capacity.
- ResolveInherits will use the node count as the suggested initial capacity.
- FromStream will now stream lines, rather than reading the whole file into memory and then splitting into lines.
These config files often contain many repeated strings which result in different string references in memory. By using a pool, we can detect when the strings are equal and reuse an existing reference as strings are immutable.
The FromLines will now use a pool to de-duplicate strings for a single call. By allowing a pool to be provided as a parameter, we can reuse even more strings. The MapCache defines such a pool so that strings are reused across all maps in the cache for even more savings.
- Add support for escaping '#' inside values
- Add support for escaping leading and trailing whitespace
And when discardCommentsAndWhitespace is set to false:
- Add proper support for comments
- Persist empty lines
Whitespace and comment support requires an explicit opt-in because
they produce MiniYamlNodes with null keys. Supporting these through
the entire game engine would require changing all yaml enumerations
to explicitly check and account for these keys with no benefit.
Comments and whitespace are now treated as real nodes during parsing,
which means that the yaml parser will throw errors if they have
incorrect indentation, even if these nodes will be discarded.
