ESP32: Using the Internal Flash File System

ESP32 File System

The ESP32 contains a lightweight filesystem specially design for microcontrollers called Serial Peripheral Interface Flash File System. This File System, also known as SPIFFS for its initials, is implemented on top of a Flash chip connected directly to the SPI bus.

SPIFFS supports basic operations on files, like opening, close, read and write, and a very rudimentary implementation of directories. You will find that several project uses this File System in order to serve pages (and files) in a Web Server.

Requirements

First of all we need to install a few dependencies. The most important is the File System uploader tool, which can be used from the Arduino IDE, Visual Studio Code, or command line. The steps for performing this installation can be found at project repository on GitHub.

If you are using Visual Studio Code, as I do, install the following extension: ESP8266FS. From the command line:

code --install-extension vscode-esp8266fs

Do not forget to install Arduino-ESP32 Core https://github.com/espressif/arduino-esp32#installation-instructions

Project configuration

When using the Arduino IDE, just selecting the board in the Board Manager will do the job. But when using Visual Studio Code, we have to set some values on .vscode/arduino.json.

{
    "sketch": "main.ino",
    "board": "esp32:esp32:featheresp32",
    "configuration": "FlashFreq=80m,UploadSpeed=921600,DebugLevel=none,PartitionScheme=default,FlashSize=4MB,FlashMode=keep",
    "port": "/dev/tty.SLAB_USBtoUART",
    "output": "output"
}

With this configuration 1M of Flash will be used for SPIFFS and the rest (3M out of 4M) for your code.

Testing the plugin configuration

In Arduino IDE using the Data Uploader is straightforward. But on Visual Studio Code the plugin should be configured carefully in order to work. The following is my Visual Studio Code configuration (settings.json) which is quite verbose for testing purposes.

{
   ...
    "esp8266fs.espota.debug": true,
    "esp8266fs.esptool.executable": "/Users/kimi/Documents/Arduino/hardware/espressif/esp32/tools/esptool.py",
    "esp8266fs.esptool.verbosity": "vvv",
    "esp8266fs.logLevel": "debug",
    "esp8266fs.mkspiffs.executable": "/Users/kimi/Documents/Arduino/hardware/espressif/esp32/tools/mkspiffs/mkspiffs",
    "esp8266fs.preferencesPath": "/Users/kimi/Library/Arduino15",
    "esp8266fs.arduinoUserPath": "/Users/kimi/Documents/Arduino",
    "esp8266fs.spiffsImage": "./spiffs.image",
    "esptool.py.spi_connection": "SPI",
    "esptool.py.compress": "true",
    "esptool.py.verify": "false",
   ...
}
ESP32 has not been installed correctly

Testing the Data Upload can be done by List SPIFFS command. cmd + shift P and select ESP8266FS: List SPIFFS. If you get an error saying

ESP32 has not been installed correctly – see https://github.com/espressif/arduino-esp32

Take into consideration that the Path of the ESP32 Tool is generated by splitting the Board key on .vscode/arduino.json in order to get three parameters: Package, Architecture and Board (ref: https://github.com/kash4kev/vscode-esp8266fs/…/esp8266fs.js#L467).

As my board is an Adafruit HUZZAH32, and my board config is

{
   ...
   "board": "esp32:esp32:featheresp32",
   ...
}

the three key parameters will be:

{
   "package": "esp32",
   "architecture": "esp32",
   "board": "featheresp32",
}

This parameters will be used by the ESP8266FS plugin to create the path where ESP tools are installed. In my setup it will result in /Users/kimi/Documents/Arduino/hardware/esp32/esp32. But, based on the ESP32 Arduino core tutorial, my hardware definition is installed on /Users/kimi/Documents/Arduino/hardware/espressif/esp32.

My workaround to this issue was to create a symbolic link so the esp32 directory was points to espressif.

ln -s espressif esp32
Couldn’t find match for argument

Another common error is Couldn’t find match for argument when running mkspiffs, which is a tool found in the ESP32 Arduino Core SDK that handles the creation, upload and download (among other responsibilities) of images.

This issue is already reported on GitHub at https://github.com/kash4kev/vscode-esp8266fs/issues/15. The workaround as explain there, is to disable the configuration esp8266fs.mkspiffs.allFiles.

Packing information into an Image

In your project directory, next to your main Arduino file, create a directory called data. All files stored in that folder will be transfer to your ESP32 microcontroller when uploading the File System.

All our data files are not transferred one by one to the ESP32, but inside an Image. In order to create that image we should pack those files using mkspiffs. The ESP8266FS extension has a command for doing that, but be can also use the command line tool.

The benefits of using the command line tool are clear, the operation can be automated inside a script.

/<Path To Arduino>/hardware/espressif/esp32/tools/mkspiffs/mkspiffs --create /<Your Arduino Project Directory>/data --size 0x30000 --page 256 --block 4096 /<Your Arduino Project Directory>/spiffs.image

Uploading files

Remember that uploading files from your Computer will replace the files present in your ESP32.

You will also find a handy command in the Visual Studio Code extension for uploading your currently created image. But if you want to do that using the command line, the following execution will do the trick.

/<Path To Arduino>/hardware/espressif/esp32/tools/esptool/esptool -vvv -ca 0x3D0000 -cd  -cp /dev/tty.SLAB_USBtoUART -cb 921600 -cf /<Your Arduino Project Directory>/spiffs.image

Where /dev/tty.SLAB_USBtoUART is the USB Serial port where I have my ESP32 connected. The result of running this command may look like

--- Uploading SPIFFS file with esptool.py ---
Python Executable: "python"
SPIFFS Uploading Image... (/Users/kimi/Works/eaceto.dev/embedded/ESP32/P1/spiffs.image)
  [SPIFFS] Python   : python
  [SPIFFS] EspTool  : /Users/kimi/Documents/Arduino/hardware/espressif/esp32/tools/esptool.py
  [SPIFFS] address  : 0x290000
  [SPIFFS] port     : /dev/tty.SLAB_USBtoUART
  [SPIFFS] speed    : 921600
  [SPIFFS] before   : default_reset
  [SPIFFS] after    : hard_reset
  [SPIFFS] flashMode: keep
  [SPIFFS] flashFreq: 80m
  [SPIFFS] flashSize: 4MB
Running: python /Users/kimi/Documents/Arduino/hardware/espressif/esp32/tools/esptool.py --chip esp32 --baud 921600 --port /dev/tty.SLAB_USBtoUART --before default_reset --after hard_reset write_flash --flash_mode keep --flash_freq 80m --flash_size 4MB 0x290000 /Users/kimi/Works/eaceto.dev/embedded/ESP32/P1/spiffs.image
esptool.py v2.8
Serial port /dev/tty.SLAB_USBtoUART
Connecting....
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, Coding Scheme None
Crystal is 40MHz
MAC: 24:0a:c4:0c:94:78
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Compressed 196608 bytes to 537...
Writing at 0x00290000... (100 %)
Wrote 196608 bytes (537 compressed) at 0x00290000 in 0.0 seconds (effective 136297.5 kbit/s)...
Hash of data verified.
Leaving...
Hard resetting via RTS pin...
ESP8266 SPIFFS interface finished.

Verifying what it’s inside an image

If you want to check which files are included in an image file, you can list them by running

<Path To Arduino>/hardware/espressif/esp32/tools/mkspiffs/mkspiffs --list --page 256 --block 4096 <Path to>/spiffs.image

Downloading an image form the ESP32 to your Computer

May be one of the most important steps when writing an image to the ESP32 is verifying that it was correctly done. For doing this you have to find how many bytes where written the uploading the image to the ESP32. This information can be found on the output of the write_flash operation. And as important as how many it is, where. This information it is also present in the same debug output.

In my previous example this two values where 196608 bytes and starting address 0x290000. So the command to execute in order to read that segment of Flash is:

<Path To Arduino>/hardware/espressif/esp32/tools/esptool.py --chip esp32 --baud 921600 --port /dev/tty.SLAB_USBtoUART read_flash  0x290000 196608 downloaded.image

Output

iFi, BT, Dual Core, Coding Scheme None
Crystal is 40MHz
MAC: 24:0a:c4:0c:94:78
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
196608 (100 %)
196608 (100 %)
Read 196608 bytes at 0x290000 in 2.5 seconds (641.6 kbit/s)...
Hard resetting via RTS pin...

Using the command explained in Verifying what’s inside the image we can see which files are inside the image

<Path To Arduino>/hardware/esp32/esp32/tools/mkspiffs/mkspiffs --list --page 256 --block 4096 downloaded.image

Output

159 /config_test.json

Next steps…

In the following post I will show how to read a JSON file and obtain a configuration parameter that will be use to configure the WiFi network to which our ESP32 will be connected.

Stay tuned!

Rollback from MicroPython to Arduino on ESP32

Arduino logo

I hope you find MicroPython is VERY useful and perfect for you. But just in case it doesn’t, or for any reason you want to rollback to Arduino. Here is how to do it.

Requisites

ESPTool (already installed if you went from Arduino to MicroPython)

pip3 install esptool

If there is a permissions problem doing this, try to run pip3 install either by using “sudo” or “–user” flag.

Steps

The following example assumes the USB port can be found at /dev/cu.SLAB_USBtoUART, but you may find it at /dev/ttyUSB0

  1. Erase your ESP32 Flash.
$ esptool.py --port /dev/cu.SLAB_USBtoUART erase_flash
esptool.py v2.8
Serial port /dev/cu.SLAB_USBtoUART
Connecting........__
Detecting chip type... ESP32
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, Coding Scheme None
Crystal is 40MHz
MAC: 24:0a:c4:0c:94:78
Uploading stub...
Running stub...
Stub running...
Erasing flash (this may take a while)...
Chip erase completed successfully in 4.1s
Hard resetting via RTS pin...

  2. Thanks it! You can now go back to Arduino IDE and build and run programs on your ESP32 board!

Another useful ESPTool’s sub command

ESPTool let us retrieve the MAC address of the device from our host. This can be used for several things, like for example creating specific setup files. In order to read the ESP32 MAC address, run ESPTool with the subcommand “read_mac” as shown below

$ esptool.py --port /dev/cu.SLAB_USBtoUART read_mac
esptool.py v2.8
Serial port /dev/cu.SLAB_USBtoUART
Connecting........__
Detecting chip type... ESP32
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, Coding Scheme None
Crystal is 40MHz
MAC: 24:0a:c4:9d:24:28
Uploading stub...
Running stub...
Stub running...
MAC: 24:0a:c4:9d:24:28
Hard resetting via RTS pin...

MicroPython on an ESP32

MicroPython

Requirements

Software on the host

ESPTools and Adafruit AMPY are required. Both can be installed using pip3. The recommended version of Python on the host is Python 3.7

pip3 install esptool --upgrade
pip3 install adafruit-ampy --upgrade

Firmware for the ESP32 module

MicroPython for ESP32. The latests firmware (compatible with ESP-IDF v4.x) when writing can be found at http://micropython.org/resources/firmware/esp32-idf4-20191220-v1.12.bin

Installation

First of all erase your ESP32 completely by running the following command. Where /dev/ttyUSB0 is the Virtual COM port created by your board.

$ esptool.py --chip esp32 -p /dev/ttyUSB0 erase_flash
esptool.py v2.8
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, Coding Scheme None
Crystal is 40MHz
MAC: 24:0a:c4:0c:94:78
Uploading stub...
Running stub...
Stub running...
Erasing flash (this may take a while)...
Chip erase completed successfully in 1.8s
Hard resetting via RTS pin...

Then, write the flash with the download image (esp32-idf4-20191220-v1.12.bin is this example).

$ esptool.py --chip esp32 -p /dev/ttyUSB0 write_flash -z 0x1000 esp32-idf4-20191220-v1.12.bin
esptool.py v2.8
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, Coding Scheme None
Crystal is 40MHz
MAC: 24:0a:c4:0c:94:78
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 1408512 bytes to 894711...
Wrote 1408512 bytes (894711 compressed) at 0x00001000 in 78.9 seconds (effective 142.9 kbit/s)...
Hash of data verified.
Leaving...
Hard resetting via RTS pin...

Testing

After successfully flashing your ESP32 module, connect it to a Serial Monitor at 115200 baud. I like to use the embedded serial monitor that Arduino IDE has.

Perform a hard reset and you should see a console output like the one below:

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
flash read err, 1000
Falling back to built-in command interpreter.
OK
>ets Jun  8 2016 00:22:57
rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:4988
load:0x40078000,len:10404
load:0x40080400,len:5680
entry 0x400806bc
 [0;32mI (519) cpu_start: Pro cpu up. [0m
 [0;32mI (519) cpu_start: Application information: [0m
 [0;32mI (519) cpu_start: Compile time:     Dec 20 2019 07:56:38 [0m
 [0;32mI (523) cpu_start: ELF file SHA256:  0000000000000000... [0m
 [0;32mI (529) cpu_start: ESP-IDF:          v4.0-beta1 [0m
 [0;32mI (534) cpu_start: Starting app cpu, entry point is 0x40083014 [0m
 [0;32mI (0) cpu_start: App cpu up. [0m
 [0;32mI (544) heap_init: Initializing. RAM available for dynamic allocation: [0m
 [0;32mI (551) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM [0m
 [0;32mI (557) heap_init: At 3FFB6388 len 00001C78 (7 KiB): DRAM [0m
 [0;32mI (563) heap_init: At 3FFB9A20 len 00004108 (16 KiB): DRAM [0m
 [0;32mI (569) heap_init: At 3FFBDB5C len 00000004 (0 KiB): DRAM [0m
 [0;32mI (575) heap_init: At 3FFCC8A0 len 00013760 (77 KiB): DRAM [0m
 [0;32mI (582) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM [0m
 [0;32mI (588) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM [0m
 [0;32mI (594) heap_init: At 40099FB8 len 00006048 (24 KiB): IRAM [0m
 [0;32mI (601) cpu_start: Pro cpu start user code [0m
 [0;32mI (619) spi_flash: detected chip: generic [0m
 [0;32mI (619) spi_flash: flash io: dio [0m
 [0;32mI (620) cpu_start: Chip Revision: 1 [0m
 [0;33mW (621) cpu_start: Chip revision is higher than the one configured in menuconfig. Suggest to upgrade it. [0m
 [0;32mI (632) cpu_start: Starting scheduler on PRO CPU. [0m
 [0;32mI (0) cpu_start: Starting scheduler on APP CPU. [0m
MicroPython v1.12 on 2019-12-20; ESP32 module with ESP32
Type "help()" for more information.
>>> 

Integrated Development Environment

Any Python IDE may be useful for programming in MicroPython (or CircuitPython if you are using an Adafruit compatible board). I personally recommend using Mu (https://codewith.mu/). Mu is programmed in Python, so it works on Windows, Linux and macOS. It also has a portable (pen drive friendly version) which it makes it ideal for students that uses university computers.

Select ESP MicroPython. This preference can be changed later.

Testing the module can be done using the Python REPL that be have already flashed. For example, importing the machine package can give us access to getting and settings properties of our board. For example, the Huzzah32 (ESP32) board runs at 240MHz, but when flashing MicroPython the clock is set at 160Mhz. That can be check running:

MicroPython v1.12 on 2019-12-20; ESP32 module with ESP32
Type "help()" for more information.
>>> import machine
>>> machine.freq()
160000000
>>> 

Increasing the CPU frequency back to 240MHz can be done calling the same function as shown below:

>>> machine.freq(240000000)    # set frequency back to 240MHz
I (1604452) pm_esp32: Frequency switching config: CPU_MAX: 240, APB_MAX: 240, APB_MIN: 240, Light sleep: DISABLED
>>> machine.freq()
240000000
>>> 

Setting up ESP32 development environment in Linux (with Arduino IDE)

Arduino logo

Installing the IDE

Arduino logo

Download Arduino IDE from https://www.arduino.cc/en/Main/Software. The latest version when writing this a article is 1.8.12 (https://www.arduino.cc/download_handler.php?f=/arduino-1.8.12-linux64.tar.xz).

Extract the content of the tar.xz file where you want to have your Arduino IDE, and then, execute install.sh. If there is an error, like:

$ ./install.sh
Adding desktop shortcut, menu item and file associations for Arduino IDE...
ln: failed to create symbolic link '/usr/local/bin/arduino': Permission denied
Adding s

try to re run the script as root

$ sudo ./install.sh 

Installing the USB drivers

There are mainly two types of USB chips that may be used as Serial to USB converters, one from FTDI and the other one from SiLabs. In order to find out which chip your ESP32 board has, disconnect the board from your PC and run lsusb.

$ lsusb
Bus 002 Device 002: ID 0bda:0411 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 046d:c534 Logitech, Inc. Unifying Receiver
Bus 001 Device 010: ID 0bda:5411 Realtek Semiconductor Corp. 4-Port USB 2.0 Hub
Bus 001 Device 004: ID 8087:0aaa Intel Corp.
Bus 001 Device 003: ID 048d:8297 Integrated Technology Express, Inc. ITE Device(8595)
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Then, connect the board and repeat the command.

$ lsusb
Bus 002 Device 002: ID 0bda:0411 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 046d:c534 Logitech, Inc. Unifying Receiver
Bus 001 Device 012: ID 10c4:ea60 Cygnal Integrated Products, Inc. CP2102/CP2109 UART Bridge Controller [CP210x family]
Bus 001 Device 010: ID 0bda:5411 Realtek Semiconductor Corp. 4-Port USB 2.0 Hub
Bus 001 Device 004: ID 8087:0aaa Intel Corp.
Bus 001 Device 003: ID 048d:8297 Integrated Technology Express, Inc. ITE Device(8595)
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The new device, in this case a CP210x (Cygnal Integrated CP2102 / CP2109 UART Bridge Controller [CP210x family], is your ESP32 board.

Bus 001 Device 012: ID 10c4:ea60 Cygnal Integrated Products, Inc. CP2102/CP2109 UART Bridge Controller [CP210x family]

Install USB driver for your board.

Adding ESP32 Boards to Arduino IDE

Arduino IDE does not support ESP32 boards (and many other 3rd party boards) by default. But adding an additional board to Arduino IDE, in this case, an ESP32 board is as easy as:

  • Start Arduino IDE and open Preferences window
  • In Additional Board Manager URLs, add the following link:
https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json
  • You can add several board definitions, separating URLs with comma
  • Open Boards Manager from Tools; Boards; and install the ESP32 platform.
  • After installation, select your ESP32 board from Tools; Board menu.