feat(build): add initial configuration for fast reflashing

Introduce ESPTOOLPY_FAST_REFLASHING configuration option. It instructs
ldgen to group entity mappings for libraries deemed mutable (prone to
change) separately from those considered immutable (unlikely to change).

Organizing mutable and immutable libraries separately in the linker
script allows the linker to form larger contiguous blocks of data for
immutable libraries in the application's output sections. These blocks
are likely to stay mostly unchanged between application recompilations,
enabling them to be skipped during reflashing.

Separating mutable and immutable libraries in the linker script to
minimize changes in the output sections is insufficient. Padding is
added after the input sections of mutable libraries in the default data
and text output sections. This creates a buffer for the mutable
libraries, allowing additional changes to be made without altering the
layout of the binary image.

Additionally two optimizations currently in use can still mix data from
these libraries, leading to significant changes even within the grouped
immutable libraries.

1. constant merging

    Linker will try to merge input sections that have the MERGE and
    STRING flags from different libraries (object files) to perform
    optimizations like tail merging. For example, adding a string
    literal in a mutable library will also change the addresses of
    string literals from immutable libraries in such a merged section,
    causing changes in the generated code when those literals are
    referenced.

    Disabled with COMPILER_DISABLE_MERGE_CONSTANTS(-fno-merge-constants)

2. literal pools on Xtensa

    As optimization, the linker may merge literal pools from different
    libraries (object files) to improve the generated code size. This
    has the same effect as constant merging, and changes in mutable
    libraries may cause changes in the generated code for immutable
    libraries. To get larger unchanged continuous blocks in the text
    output sections for immutable libraries, we need to ensure that the
    Xtensa literal pools remain close to their references and are not
    merged.

    Disabled with CONFIG_COMPILER_ENABLE_TEXT_SECTION_LITERALS(-mtext-section-literals)

When ESPTOOLPY_FAST_REFLASHING is enabled, these two optimizations are
disabled to achieve larger unchaged continuous blocks for the grouped
immutable libraries, even though disabling these optimizations results
in slightly larger code.

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>

CMakeLists.txt

@@ -159,6 +159,10 @@ if(CONFIG_COMPILER_NO_MERGE_CONSTANTS)
     list(APPEND compile_options "-fno-merge-constants")
 endif()
 
+if(CONFIG_COMPILER_ENABLE_TEXT_SECTION_LITERALS)
+    list(APPEND compile_options "-mtext-section-literals")
+endif()
+
 if(CONFIG_COMPILER_STACK_CHECK_MODE_NORM)
     list(APPEND compile_options "-fstack-protector")
 elseif(CONFIG_COMPILER_STACK_CHECK_MODE_STRONG)

Kconfig

@@ -547,6 +547,20 @@ mainmenu "Espressif IoT Development Framework Configuration"
                 distribution is more uniform across libraries. On downside, it may increase
                 the binary size and hence should be used during development phase only.
 
+        config COMPILER_ENABLE_TEXT_SECTION_LITERALS
+            bool
+            depends on IDF_TOOLCHAIN_GCC
+            depends on IDF_TARGET_ARCH_XTENSA
+            default y if ESPTOOLPY_FAST_REFLASHING
+            help
+                Intersperse Xtensa literals within the text section to keep
+                them as close as possible to their references. This prevents
+                literals from being placed into a separate section in the
+                output file and prevents the linker from combining literal
+                pools from different object files. Enabling this is necessary
+                for fast reflashing to prevent mixing code from mutable and
+                immutable libraries.
+
         config COMPILER_WARN_WRITE_STRINGS
             bool "Enable -Wwrite-strings warning flag"
             default "n"
@@ -755,3 +769,4 @@ mainmenu "Espressif IoT Development Framework Configuration"
             - CONFIG_GDMA_ENABLE_WEIGHTED_ARBITRATION
             - CONFIG_I3C_MASTER_ENABLED
             - CONFIG_MBEDTLS_ESP_IDF_USE_PSA_CRYPTO
+            - CONFIG_ESPTOOLPY_FAST_REFLASHING

components/esptool_py/Kconfig.projbuild

@@ -202,4 +202,65 @@ menu "Serial flasher config"
         int
         default ESP_CONSOLE_UART_BAUDRATE if ESP_CONSOLE_UART
         default 115200 if !ESP_CONSOLE_UART
+
+    menu "Fast Reflashing"
+
+        config ESPTOOLPY_FAST_REFLASHING
+            bool "Enable fast reflashing (Experimental)"
+            depends on IDF_EXPERIMENTAL_FEATURES
+            select COMPILER_NO_MERGE_CONSTANTS if IDF_TOOLCHAIN_GCC
+            help
+                Enabling this option classifies component libraries into two
+                categories: mutable and immutable. Mutable libraries are
+                expected to change frequently during development, while
+                immutable libraries are considered stable. All project
+                component libraries are treated as mutable; all other libraries
+                are treated as immutable.
+
+                In the generated linker script, input sections from immutable
+                libraries are placed before those from mutable libraries. This
+                layout helps localize changes in the output sections of the ELF
+                file, so that recompilation primarily affects a confined area
+                associated with the mutable libraries.
+
+                This enables the generation of binary images with large
+                unmodified regions across recompilations, allowing to flash
+                only the parts that have changed.
+
+                To support this, some compiler optimizations, such as constant
+                merging, are disabled, which helps minimize differences in the
+                binary image between builds.  However, this may increase the
+                overall image size and flash usage.
+
+                Additionally, this option inserts a padding, defined by the
+                ESPTOOLPY_FAST_REFLASHING_PADDING option, after the input
+                sections of mutable libraries. This allows further isolate
+                changes but also increases the size of the binary image and
+                flash consumption.
+
+        choice ESPTOOLPY_FAST_REFLASHING_PADDING_SIZE
+            prompt "Padding size"
+            default ESPTOOLPY_FAST_REFLASHING_PADDING_4KB
+            depends on ESPTOOLPY_FAST_REFLASHING
+            help
+                Selects the amount of padding (in kilobytes) to add after input
+                sections of mutable libraries when fast reflashing is enabled.
+
+            config ESPTOOLPY_FAST_REFLASHING_PADDING_2KB
+                bool "2 KB"
+            config ESPTOOLPY_FAST_REFLASHING_PADDING_4KB
+                bool "4 KB"
+            config ESPTOOLPY_FAST_REFLASHING_PADDING_8KB
+                bool "8 KB"
+        endchoice
+
+        config ESPTOOLPY_FAST_REFLASHING_PADDING
+            int
+            depends on ESPTOOLPY_FAST_REFLASHING
+            default 2048 if ESPTOOLPY_FAST_REFLASHING_PADDING_2KB
+            default 4096 if ESPTOOLPY_FAST_REFLASHING_PADDING_4KB
+            default 8192 if ESPTOOLPY_FAST_REFLASHING_PADDING_8KB
+
+    endmenu
+
 endmenu