Frantisek Hrbata
e4fa15b50a
Profiling shows that parsing the linker fragment (.lf) files with
pyparsing is the single largest cost in ldgen (~58% of total runtime).
parse_fragment_file() rebuilds its grammar from scratch on every call
because the conditional parser closure has to capture sdkconfig to
evaluate if/elif/else at parse time.
Add a cache that pickles the parsed FragmentFile object list to
<output>.lfcache. The cache is keyed on fragment file paths+mtimes
plus sdkconfig and kconfig mtimes — sdkconfig is included because
fragment if/elif/else blocks are evaluated against it at parse time.
This avoids the pyparsing cost on rebuilds where ldgen has to run
but the fragment files themselves are unchanged — for example when
a function is added or removed in a source file. Cache hits produce
the same sections.ld as a full parse, and any pickle load failure
falls through to a normal parse, so cache corruption or a Python
upgrade cannot produce a wrong build.
Print "Skipping linker fragment parsing, fragment files unchanged"
on the lf cache hit path so build logs show when the optimization
fired — this makes it easy to diagnose user reports of unexpected
ldgen behavior.
The LDGEN_NO_CACHE environment variable disables the lf cache. When
set, ldgen falls through to a normal parse without consulting or
writing the cache file.
Measured on wifi_station (205 libs, 67 .lf files), median of 10 runs:
before after
rebuild, section change, .lf unchanged 5.82s 1.34s
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Linker Script Generator
Contains code that implements linker script generation, ldgen. For more information about the feature,
see docs/en/api-guides/linker-script-generation.rst.
Source Files
The following are the source files in the directory:
ldgen.py- Python executable that gets called during build.entity.py- contains classes related to entities (library, object, symbol or combination of the above) with mappable input sections.fragments.py- contains classes for parsing the different types of fragments in linker fragment files.generation.py- contains bulk of the logic used to process fragments into output commands.sdkconfig.py- used for evaluating conditionals in fragment files.linker_script.py- augments the input linker script template with output commands from generation process to produce the output linker script.output_commands.py- contains classes that represent the output commands in the output linker script.ldgen_common.py- contains miscellaneous utilities/definitions that can be used in the files mentioned above.
Tests
Unit tests are in the test directory. These tests are run as part of CI in the job test_ldgen_on_host.
There is also a test app for ldgen in tools/test_apps/build_system/ldgen_test.
Build System
Linker script generation is a part of the build process. The build scripts tools/cmake/ldgen.cmake
and make/ldgen.mk contain the build-system-side implementation for CMake and Make, respectively.
Basic Flow
The build system invokes ldgen.py, passing some information from the build.
The linker fragment files are parsed by fragments.py, evaluating conditional expressions
with sdkconfig.py.
From the parsed fragments, generation.py generates output commands defined in output_commands.py,
with some help from entity.py.
linker_script.py writes the output linker script, replacing markers with output commands generated.
More details about the implementation are in the respective source files.