Testing MakeCode

To test MakeCode, the process is slightly more involved, due to the web-based nature and many stages of compilation, in order to provide the user with the ability to compile in-the-browser.

To conduct tests accurately, you will need to perform a few modifications and intricate compile steps. These will be documented as clearly as possible.

The tests we conducted are listed below.

Warning

Please ignore any warnings from the MakeCode editor when running these tests. We didn't add required simulations for new code we added to conduct these tests.

GPIO toggle time

This test is used to determine the time taken to toggle the GPIO. This time is then used as an offset in other other tests to remove this delay from our results.

Steps to reproduce

  1. Destroy any running local makecode instances by killing the server, and closing the browser.
  2. Ensure ./makecode/pxt-microbit/libs/core/pxt.json has debug and heap_debug set to 0.
  3. Change to the ./makecode/pxt-microbit directory.
  4. Run pxt serve and wait until a web browser opens with a local instance of MakeCode.
  5. Copy the contents of root/tests/makecode/gpio-base/GPIO-test.ts into the editor window.
  6. Hit download to download the compiled program as a hex file.
  7. Drag the file using the files GUI or use the cp command to copy the hex file to the micro:bit
  8. Use an oscilloscope set to 10us per division and measure the output.

Test code

function set_gpio(state: number) {
    pins.digitalWritePin(DigitalPin.P1, state)
}

while (true) {
    set_gpio(1);
    set_gpio(0);
}

Where is this result used?

In each result where time is reported.

Context switch time

This test determines the cost of our stack duplication approach with respect to time in MakeCode. We page between two fibers, and observe the impact this has on execution time using GPIO and an oscilloscope.

Steps to reproduce

  1. Destroy any running local makecode instances by killing the server, and closing the browser.
  2. Ensure ./makecode/pxt-microbit/libs/core/pxt.json has debug and heap_debug set to 0.
  3. Change to the ./makecode/pxt-microbit directory.
  4. Run pxt serve and wait until a web browser opens with a local instance of MakeCode.
  5. Copy the contents of root/tests/makecode/context-switch-time/context-switch-test.ts into the editor window.
  6. Hit download to download the compiled program as a hex file.
  7. Drag the file using the files GUI or use the cp command to copy the hex file to the micro:bit
  8. Use an oscilloscope set to 25us per division and measure the output.

Do not forget to subtract the time taken to toggle a GPIO from these results.

Test code

function set_gpio(state: number) {
    pins.digitalWritePin(DigitalPin.P1, state)
}

function high() {
    while (1) {
        set_gpio(1);
        control.spinScheduler();
    }
}

function low() {
    while (1) {
        set_gpio(0);
        control.spinScheduler();
    }
}

set_gpio(0);

control.inBackground(high)
control.inBackground(low)

control.releaseFiber();

Where is this result used?

Figure 7, where we report context switch time vs. stack size, and Figure 6, where we report the context switch profiles for each device.

Tight loop execution time

This test counts from 0 to 100,000, using the higher level language in MakeCode, TypeScript. After a full iteration, a GPIO is toggled, allowing us to calculate the time using an oscilloscope.

Steps to reproduce

  1. Destroy any running local makecode instances by killing the server, and closing the browser.
  2. Ensure ./makecode/pxt-microbit/libs/core/pxt.json has debug and heap_debug set to 0.
  3. Change to the ./makecode/pxt-microbit directory.
  4. Run pxt serve and wait until a web browser opens with a local instance of MakeCode.
  5. Copy the contents of root/tests/makecode/tight-loop/tight-loop-test.ts into the editor window.
  6. Hit download to download the compiled program as a hex file.
  7. Drag the file using the files GUI or use the cp command to copy the hex file to the micro:bit
  8. Use an oscilloscope set to 25us per division and measure the output.

Test code

function set_gpio(state: number) {
    pins.digitalWritePin(DigitalPin.P1, state)
}

while (true) {
    set_gpio(0);
    for (let i = 0; i < 100000; i++) {
        i = i;
    }
    set_gpio(1);
    for (let i = 0; i < 100000; i++) {
        i = i;
    }
}

Where is this result used?

Table 2, where we report the execution speed of each environment compared to MakeCode and Codal.

Memory consumption

This test uses the map file generated by PXT to break down all libraries into individual units, and computes the RAM / Flash cost of each.

Steps to reproduce

  1. Ensure ./makecode/pxt-microbit/libs/core/pxt.json has debug and heap_debug set to 0 and was built previously with these settings.
  2. Change to the ./makecode/pxt-microbit directory.
  3. Run node ../../tests/makecode/memory-consumption/map-file-stats.js ./libs/blocksprj/built/yt/build/bbc-microbit-classic-gcc/source/pxt-microbit-app.map in your terminal.
  4. Compute the results. Results with 'RAM' in front, indicate pre-allocated RAM in the globals or bss space. Results without 'RAM' in front are flash sizes. Both are reported in kB.

Example

Given the following memory map:

root@debian-lctes:/home/lctes/evaluators/makecode/pxt-microbit# node ../../tests/makecode/memory-consumption/map-file-stats.js ./libs/blocksprj/built/yt/build/bbc-microbit-classic-gcc/source/pxt-microbit-app.map
TOTAL 94.342
RAM.TOTAL 170.707
RAM.xr 160
RAM.xrw 8
lib/gcc/arm-none-eabi/7.2.1/thumb/v6-m/crtbegin.o 0.07
lib/thumb/v6-m/crt0.o 0.117
libs/blocksprj/built/yt/source/core/pins.cpp.o 0.852
libs/blocksprj/built/yt/source/core/serial.cpp.o 0.383
libs/blocksprj/built/yt/source/core/pxt.cpp.o 3.877
libs/blocksprj/built/yt/source/core/images.cpp.o 0.365
libs/blocksprj/built/yt/source/core/control.cpp.o 0.137
libs/blocksprj/built/yt/source/core/core.cpp.o 1.396
libs/blocksprj/built/yt/source/core/ManagedBuffer.cpp.o 0.74
libs/blocksprj/built/yt/source/core/led.cpp.o 0.27
libs/blocksprj/built/yt/source/pointers.cpp.o 0.785
libs/blocksprj/built/yt/source/radio/radio.cpp.o 1.471
libs/blocksprj/built/yt/source/main.cpp.o 0.027
libs/blocksprj/built/yt/source/core/input.cpp.o 0.623
libs/blocksprj/built/yt/source/core/basic.cpp.o 0.432
ym/microbit/source/microbit.a 1.063
ym/microbit-dal/source/microbit-dal.a 33.063
ym/ble/source/ble.a 0.431
ym/ble-nrf51822/source/ble-nrf51822.a 9.088
ym/nrf51-sdk/source/nrf51-sdk.a 7.268
ym/mbed-classic/existing/mbed-classic.a 7.346
libstdc++_nano.a 0.459
libgcc.a 12.531
libm.a 7.418
libc_nano.a 3.463
libnosys.a 0.031
libs/blocksprj/built/yt/source/core/buffer.cpp.o 0.639
RAM.lib/gcc/arm-none-eabi/7.2.1/thumb/v6-m/crtbegin.o 0.031
RAM.ym/microbit-dal/source/microbit-dal.a 0.214
RAM.ym/nrf51-sdk/source/nrf51-sdk.a 0.461
RAM.ym/mbed-classic/existing/mbed-classic.a 0.251
RAM.libc_nano.a 0.461
RAM.ym/ble-nrf51822/source/ble-nrf51822.a 0.191
RAM.libm.a 0.001
RAM.libs/blocksprj/built/yt/source/core/pxt.cpp.o 1.039
RAM.libs/blocksprj/built/yt/source/core/pins.cpp.o 0.008
RAM.libs/blocksprj/built/yt/source/radio/radio.cpp.o 0.022
RAM.ym/ble/source/ble.a 0.02
RAM.libstdc++_nano.a 0.004
RAM.libnosys.a 0.004

We can derive the following:

libs/blocksprj/built/yt/source/core/pins.cpp.o 0.852
libs/blocksprj/built/yt/source/core/serial.cpp.o 0.383
libs/blocksprj/built/yt/source/core/pxt.cpp.o 3.877
libs/blocksprj/built/yt/source/core/images.cpp.o 0.365
libs/blocksprj/built/yt/source/core/control.cpp.o 0.137
libs/blocksprj/built/yt/source/core/core.cpp.o 1.396
libs/blocksprj/built/yt/source/core/ManagedBuffer.cpp.o 0.74
libs/blocksprj/built/yt/source/core/led.cpp.o 0.27
libs/blocksprj/built/yt/source/pointers.cpp.o 0.785
libs/blocksprj/built/yt/source/radio/radio.cpp.o 1.471
libs/blocksprj/built/yt/source/main.cpp.o 0.027
libs/blocksprj/built/yt/source/core/input.cpp.o 0.623
libs/blocksprj/built/yt/source/core/basic.cpp.o 0.432
libs/blocksprj/built/yt/source/core/buffer.cpp.o 0.639

Summed is 11.997, and relates to “MakeCode” in Table 3.

ym/microbit/source/microbit.a 1.063
ym/microbit-dal/source/microbit-dal.a 33.063

Summed is 34.126, and relates to “CODAL” in Table 3.

ym/ble/source/ble.a 0.431
ym/ble-nrf51822/source/ble-nrf51822.a 9.088
ym/nrf51-sdk/source/nrf51-sdk.a 7.268
ym/mbed-classic/existing/mbed-classic.a 7.346

Summed is 24.133, and relates to “Supporting Libraries” in Table 3.

libstdc++_nano.a 0.459
libgcc.a 12.531
libm.a 7.418
libc_nano.a 3.463
libnosys.a 0.031
lib/gcc/arm-none-eabi/7.2.1/thumb/v6-m/crtbegin.o 0.07
lib/thumb/v6-m/crt0.o 0.117

Summed is 24.089, and relates to “C++ Standard Library” in Table 3.

RAM.libs/blocksprj/built/yt/source/core/pxt.cpp.o 1.039
RAM.libs/blocksprj/built/yt/source/core/pins.cpp.o 0.008
RAM.libs/blocksprj/built/yt/source/radio/radio.cpp.o 0.022

Summed is 1.069, and relates to “MakeCode” in Table 4.

RAM.ym/microbit-dal/source/microbit-dal.a 0.214

Summed is 0.214, and relates to “CODAL” in Table 4.

RAM.ym/nrf51-sdk/source/nrf51-sdk.a 0.461
RAM.ym/mbed-classic/existing/mbed-classic.a 0.251
RAM.ym/ble-nrf51822/source/ble-nrf51822.a 0.191
RAM.ym/ble/source/ble.a 0.02

Summed is 0.923, and relates to “Supporting Libraries” in Table 4.

RAM.lib/gcc/arm-none-eabi/7.2.1/thumb/v6-m/crtbegin.o 0.031
RAM.libc_nano.a 0.461
RAM.libm.a 0.001
RAM.libstdc++_nano.a 0.004
RAM.libnosys.a 0.004

Summed is 0.501, and should relate to “C++ Standard Library” in Table 4. The outlier in this result is libc_nano, which we originally found to be 0.109kb. The discrepancies in the final flash and RAM sizes calculated is down to the older version of gcc that is running on the VM. We compiled using arm-none-eabi-gcc v7.02.

Where is this result used?

Tables 4 and 5, where we report the flash and RAM usage of individual compilation units.

Context switch stack usage

This test is used to determine the default stack depth in MakeCode, and thus it's contribution to the time taken to page out the stack.

Steps to reproduce

  1. Destroy any running local makecode instances by killing the server, and closing the browser.
  2. Set debug and heap_debug in ./makecode/pxt-microbit/libs/core/pxt.json to 1.
  3. Change to the ./makecode/pxt-microbit directory.
  4. Run pxt serve and wait until a web browser opens with a local instance of MakeCode.
  5. Copy the contents of root/tests/makecode/context-switch-time/context-switch-test.ts into the editor window.
  6. Hit download to download the compiled program as a hex file.
  7. Drag the file using the files GUI or use the cp command to copy the hex file to the micro:bit
  8. In minicom (described here) the output should read sd: xxx bufferSize: xx, the former shows the default stack depth in bytes for each fiber, the latter shows the amount of bytes allocated to contain the stack.

Test code

function set_gpio(state: number) {
    pins.digitalWritePin(DigitalPin.P1, state)
}

function high() {
    while (1) {
        set_gpio(1);
        control.spinScheduler();
    }
}

function low() {
    while (1) {
        set_gpio(0);
        control.spinScheduler();
    }
}

set_gpio(0);

control.inBackground(high)
control.inBackground(low)

control.releaseFiber();

Where is this result used?

Figure 6, where we show the context switch time per device, broken down by environment. We calculate each segment (Codal, Stack, MakeCode) using the time data gathered in the context switch time test. We then use codal as the benchmark to compute the impact of the stack for each; subtracting codal, and stack page times, gives us the overhead of MakeCode.