STM32 and FreeRTOS: Finding the Cause of Hard Faults, Hangs and Random Resets

A fault is information, if you capture it

A device that resets at random, hangs after a day or stops in the middle of a transfer is one of the most expensive problems in firmware, mostly because the evidence disappears with the reset. The first rule is to stop losing it. On a Cortex-M part such as an STM32, every serious crash passes through a fault handler, and the processor has already saved the state that explains it. The job is to read that state before anything overwrites it.

Step 1: capture the registers at a hard fault

The default hard fault handler is an empty loop, which tells you nothing. Replace it with a handler that finds the stack that was in use, and passes it to a function that can read the saved registers and the fault status registers.

__attribute__((naked)) void HardFault_Handler(void)
{
    __asm volatile (
        "tst lr, #4\n"
        "ite eq\n"
        "mrseq r0, msp\n"
        "mrsne r0, psp\n"
        "b hard_fault_c\n");
}

void hard_fault_c(uint32_t *stack)
{
    volatile uint32_t pc    = stack[6];   /* where it happened */
    volatile uint32_t lr    = stack[5];   /* who called */
    volatile uint32_t cfsr  = SCB->CFSR;  /* what kind of fault */
    volatile uint32_t hfsr  = SCB->HFSR;
    volatile uint32_t mmfar = SCB->MMFAR; /* bad address, if valid */
    volatile uint32_t bfar  = SCB->BFAR;
    (void)pc; (void)lr; (void)cfsr; (void)hfsr; (void)mmfar; (void)bfar;
    for (;;) { }   /* set a breakpoint here, or store the values and reset */
}

The program counter value points at the instruction that faulted; look it up in the map file or the disassembly. The configurable fault status register says whether it was an undefined instruction, an unaligned access, a bus error on a data access or a memory management violation, and the address registers say which address. For a product in the field, write these values to a reserved RAM area that survives a reset, or to flash, and report them on the next start: a one-line crash record is worth more than a week of guessing.

Step 2: the interrupt priority trap

This is the most common FreeRTOS fault on STM32 and it produces hangs and hard faults that appear days apart. On Cortex-M, a smaller priority number means a higher priority, and FreeRTOS defines a threshold, configMAX_SYSCALL_INTERRUPT_PRIORITY. An interrupt that calls any FreeRTOS function, even a "FromISR" one, must have a priority number equal to or greater than that threshold (logically at or below it). An interrupt above the threshold may not touch the kernel at all.

• Set the interrupt priority grouping to use all bits for preemption priority (on the STM32 HAL this is the 4-bit grouping), as FreeRTOS expects.
• Check every peripheral interrupt that calls xQueueSendFromISR, xSemaphoreGiveFromISR or similar: its priority must be at or below the threshold.
• Never call a non-ISR version of a FreeRTOS function from an interrupt. Turn on configASSERT; the kernel then checks the priority for you and stops at the line that is wrong.

Step 3: stack overflow

Each task has its own stack, and in FreeRTOS the size you pass when creating a task is counted in words, not bytes, a classic source of stacks that are four times too small. A task that overflows corrupts whatever lies below it, and the crash happens later and elsewhere.

• Enable stack overflow checking in the configuration (method 2 is the more thorough) and provide the hook function; make it record the task name.
• Read each task's high-water mark with uxTaskGetStackHighWaterMark during a realistic run and size stacks with margin. Printing a few formatted lines, using a file system or calling deep library code needs far more stack than a simple task.
• On parts with a floating-point unit, a task that uses floating point saves more state, and its stack is larger.
• The main stack, used by interrupts and startup, has its own size in the linker script. An overflow there looks like a random fault.

Step 4: heap exhaustion and fragmentation

If tasks, queues and buffers are created and deleted at run time, the heap fragments and eventually an allocation fails, usually at a moment you did not test. Define the malloc-failed hook, log free-heap statistics over time, and prefer static allocation for anything created once. Dynamic creation of the long-lived objects at start-up, and never afterwards, removes the whole class.

Step 5: memory placement and DMA

Some STM32 families have memory regions that the processor can use but DMA cannot reach. The F4 series, for example, has core-coupled RAM, fast and convenient for stacks and variables, but not accessible by DMA; a buffer placed there makes a transfer fail silently or fault. Check the linker script for where the stacks and the DMA buffers actually land. On parts with a data cache, a DMA buffer must also be cache-line aligned and be cleaned or invalidated around the transfer, or the processor sees stale data. In our STM32F407 industrial control project, which combined FreeRTOS, a FAT file system on an SD card through SDIO and external SRAM, buffer placement and task stack sizes were what decided stability under sustained recording.

Step 6: shared data, priority inversion and deadlock

• Data shared between an interrupt and a task needs protection even if it is "just a flag". Use a queue, a task notification or a critical section, and mark shared variables correctly.
• Use a mutex, not a binary semaphore, to protect a resource shared by tasks of different priority, since a mutex has priority inheritance and a semaphore does not. Otherwise a low-priority task holding the resource can block a high-priority one for as long as a medium-priority task keeps running.
• Two tasks taking two locks in opposite orders will deadlock eventually. Fix the lock order, or use one lock.
• Do not block inside a critical section, and keep interrupt handlers short: defer work to a task.

Step 7: a watchdog that explains the reset

An independent watchdog turns a hang into a reset; the point is to make the reset informative. Do not feed the watchdog from a timer interrupt or from one low-priority loop, because then it proves only that the timer works. Have every important task report in with a flag or an event bit, and feed the watchdog from one supervisor task only when all of them have. When it is not satisfied, record which task was missing before the reset occurs. On the next start, read the reset-cause register (watchdog, brown-out, software, pin) and log it with the saved fault data.

Step 8: trace and test

When the cause is a timing interaction rather than a single bad pointer, a trace of task switches and interrupts is what you need. Many STM32 parts support instruction and event trace through the debug port, and RTOS-aware trace tools show which task ran, which interrupt preempted it and what the kernel was waiting for. A hardware test that runs the system for days at temperature, with deliberate stress on communication and storage, catches faults that a short bench test never will.

Where we can help

We write and rescue STM32 firmware under FreeRTOS and bare metal, including long-running industrial and low-power products. See our firmware development service and our guide to embedded software architecture and RTOS design.