During the merge operation, it is quite common to be dealing with a node that has no child nodes. When there are no such nodes, we can return early from some functions to avoid allocating new collections that will not be used.
In the MergePartial operation, reuse a dictionary as scratch space when checking for conflicts. We introduce a IntoDictionaryWithConflictLog helper to allow this. This avoids allocating a new dictionary for the conflict log that gets thrown away at each check.
Provide an explicit size of 512, smaller than the default 2048. The cursors for all mods still pack onto this single sheet, so we avoid wasting memory on larger sheets.
Sorting the cursors also helps pack them onto the sheets more efficiently.
Sheets can be buffered - where a copy of their data is kept in RAM. This allows for modifications to the sheet to be batched in the RAM buffer, before later being committed to VRAM in a single operation. The SheetBuilder class allows for sprites to be allocated onto one or more sheets. Each time a sheet is filled, it will allocate a new sheet. Sheets allocated by the sheet builder will typically use the buffering mechanism.
Previously each time the builder allocated a new sheet, the buffer would get thrown away, and the next sheet would allocate a fresh buffer. These buffers can be large and may accumulate before the GC cleans them up. So although only one buffer will be live at a time, they can cause a spike in memory used by the process during loading.
We can avoid this issue by allowing the buffer from the previous sheet to be reused by the next sheet. This is possible because the sheet builder only has one live sheet for modifications at a time, and they are all the same type and size. By allocating only one buffer per builder instead of one per sheet, we reduce the peak memory required during loading.
- In CursorManager, we can release the buffer on the final sheet after loading cursors. We don't need to release the buffer on every sheet in the builder, as the builder will handle that.
- In SpriteCache, we don't need to call CreateBuffer explicitly, as the builder will do that for us.
- In RadarWidget, we don't need to call CreateBuffer explicitly, as GetData will do that for us.
The default Stream implementation of this method has to rent an array so it can call the overload that accepts an array, and then copy the output over. This is because the array overload is required and the span overload was only added more recently.
We can avoid the overhead of this by implementing the span overload and working with the destination span directly. Do so for all classes we have that derive from Stream, and redirect their array overload to the span overload for code reuse.
The ExtractEmmyLuaAPI utility command, invoked with `--emmy-lua-api`, produces a documentation file that is used by the [OpenRA Lua Language Extension](https://marketplace.visualstudio.com/items?itemName=openra.vscode-openra-lua) to provide documentation and type information is VSCode and VSCode compatible editors when editing the Lua scripts.
We improve the documentation and types produced by this utility in a few ways:
- Require descriptions to be provided for all items.
- Fix the type definitions of the base engine types (cpos, wpos, wangle, wdist, wvec, cvec) to match with the actual bindings on the C# side. Add some extra bindings for these types to increase their utility.
- Introduce ScriptEmmyTypeOverrideAttribute to allow the C# side of the bindings to provide a more specific type. The utility command now requires this to be used to avoid accidentally exporting poor type information.
- Fix a handful of scripts where the new type information revealed warnings.
The ability to ScriptEmmyTypeOverrideAttribute allows parameters and return types to provide a more specific type compared to the previous, weak, type definition. For example LuaValue mapped to `any`, LuaTable mapped to `table`, and LuaFunction mapped to `function`. These types are all non-specific. `any` can be anything, `table` is a table without known types for its keys or values, `function` is a function with an unknown signature.
Now, we can provide specific types. , e.g. instead of `table`, ReinforcementsGlobal.ReinforceWithTransport is able to specify `{ [1]: actor, [2]: actor[] }` - a table with keys 1 and 2, whose values are an actor, and a table of actors respectively. The callback functions in MapGlobal now have signatures, e.g. instead of `function` we have `fun(a: actor):boolean`. In UtilsGlobal, we also make use of generic types. These work in a similar fashion to generics in C#. These methods operate on collections, we can introduce a generic parameter named `T` for the type of the items in those collections. Now the return type and callback parameters can also use that generic type. This means the return type or callback functions operate on the same type as whatever type is in the collection you pass in. e.g. Utils.Do accepts a collection typed as `T[]` with a callback function invoked on each item typed as `fun(item: T)`. If you pass in actors, the callback operates on an actor. If you pass in strings, the callback operates on a string, etc.
Overall, these changes should result in an improved user experience for those editing OpenRA Lua scripts in a compatible IDE.
Currently when linting translations, we check for any unused translation keys. This works fine for the default mods, which own the entire sets of translation files. For community mods, they often import the translation files from the common mod assets, but they may only use some of the translations provided. Currently, they would get warnings about not using translations from the common files they have imported.
Since the community mods don't own those translations, getting warnings about it is annoying. To solve this issue, introduce a AllowUnusedTranslationsInExternalPackages in the mod.yaml which defaults to true. This will prevent reporting of unused translation keys from external assets. Keys that are used for external assets will still be validated, and keys from the mod assets will be both validated and unused keys will be reported.
We default the new flag to true and don't provide an update rule. This means community mods will get the new behaviour. For the default mods, we do want to check the "external" assets, since we control those assets. So the default mods have their mod.yaml updated to disable the flag and retain the existing behaviour of checking everything.
- Previously the Inherits syntax was only resolved when used for top-level nodes. Now it is also resolved for nested nodes as well.
- Previously the MiniYAML Merge feature supported the ability to remove nodes, but this only worked within the context of inherited nodes. Now, we allow node removal to work outside of the inheritance context.
- EditorActorLayer now tracks previews on map with a SpatiallyPartitioned instead of a Dictionary. This allows the copy-paste logic to call an efficient PreviewsInCellRegion method, instead of asking for previews cell-by-cell.
- EditorActorPreview subscribes to the CellEntryChanged methods on the map. Previously the preview was refreshed regardless of which cell changed. Now the preview only regenerates if the preview's footprint has been affected.
- Use error.Message to reports errors, as error.ToString isn't overridden.
- Ensure multiple translation languages are handled correctly, we need to use a list to maintain English being the first one.
For indexed PNGs, we only need to allocate a palette large enough to accommodate the number of indexed colours in the image. For images that don't use all 256 colours, this slightly reduces the memory usage for these images.
When LoadFromManifest is called, trim the various backing collections. These backing collections tend to live a long time without further modifications.
As the SpriteCache is used as a one-shot operation in practise, holding on to the capacity of backing collections is not required. Memory usage can be reduced by allowing the capacity to be reset after the SpriteCache has resolved items.
- Once LoadReservations is called, reset the reservation dictionaries so their backing collections can be reclaimed.
- When ResolveSprites is called, shrink the resolved dictionary as resolutions take place.
In a3d0a50f4d, SpriteCache is updated to sort sprites by height before adding them onto the sheet. This improves packing by reducing wasted space as the sprites are packed onto the sheet. D2kSpriteSequence does not fully benefit from this change, as it creates additional sprites afterwards in the ResolveSprites method. These are not sorted, so they often waste space due to height changes between adjacent sprites and cause an inefficient packing. Sorting them in place is insufficient, as each sequence performs the operation independently. So sets of sprites across different sequences end up with poor packing overall. We need all the sprites to be collected together and sorted in one place for best effect.
We restructure SpriteCache to allow a frame mutation function to be provided when reserving sprites. This removes the need for the ReserveFrames and ResolveFrames methods in SpriteCache. D2kSpriteSequence can use this new function to pass in the required modification, and no longer has to add frames to the sheet builder itself. Now the SpriteCache can apply the desired frame mutations, it can batch together these mutated frames with the other frames and sort them all as a single batch. With all frames sorted together the maximum benefit of this packing approach is realised.
This reduces the number of BGRA sheets required for the d2k mod from 3 to 2.
When sheet builders are adding sprites to a sheet, they work left to right along each row. They reserve height for the highest sprite seen along that row, resetting the height reservation when the row runs out of space and it moves down to the next row.
As the SpriteCache adds the sprites in a giant batch, it can optimise this operation by ordering the sprites by their height. This reduces wastage where shorter sprites don't use the the full height reserved within the row. The reduced wastage can help the sheet builder allocate fewer sheets, improving load times and improving GPU memory usage as less texture memory is required.
The assets for the Tiberian Dawn HD mod are much larger than assets for the default mods, causing a lot of load time to be spent in Util.FastCopyIntoChannel.
We can provide a special case for the SpriteFrameType.Bgra32 format, which is the same format as the destination buffer. In this scenario we can just perform memory copies between the source and destination. Additionally, whilst the default mods require all their assets to get their alpha premultiplied, many of the Tiberian Dawn assets are already premultiplied. Being able to skip this step for these assets saves us having to interpret the bytes into colors and back again.
For the default mods, there isn't a noticeable timing difference. For Tiberian Dawn HD or other mods with modern assets sizes, a large speedup is achieved.
This is a more natural representation than int that allows removal of casts in many places that require uint. Additionally, we can change the internal representation from long to uint, making the Color struct smaller. Since arrays of colors are common, this can save on memory.
- Rename the filename parameter to name and make it mandatory. Review all callers and ensure a useful string is provided as input, to ensure sufficient context is included for logging and debugging. This can be a filename, url, or any arbitrary text so include whatever context seems reasonable.
- When several MiniYamls are created that have similar content, provide a shared string pool. This allows strings that are common between all the yaml to be shared, reducing long term memory usage. We also change the pool from a dictionary to a set. Originally a Dictionary had to be used so we could call TryGetValue to get a reference to the pooled string. Now that more recent versions of dotnet provide a TryGetValue on HashSet, we can use a set directly without the memory wasted by having to store both keys and values in a dictionary.
