Edu-Pencil

Air Pencil

Learning by Doing

Checkout the demo

Demo

AIR PENCIL is a part of the TensorFlow Lite for Microcontroller Experiments Challenge, adding to a collection of open source, interactive projects designed to demonstrate some fun ways to combine Arduino and TensorFlow Lite for Microcontrollers.

These projects were built with the

Other experiments to explore:

Tools

Flashing Arduino Nano

Flashing: Using the Arduino Nano Sense 33 BLE

  1. Install the Arduino IDE
  2. Setup Arduino board:
    • Plug in the board
    • Install the board by navigating to Tools > Board > Boards Manager and search for Arduino Mbed OS Nano Boards. Full instructions (including drivers required for Windows) here.
    • FAQ for connection problems can be found here.
    • After the board is installed, select it under to Tools > Board > Arduino Mbed OS Nano Boards > Arduino Nano 33 BLE Arduino board
  1. Install Arduino libraries
    • Navigate to Tools > Manage Libraries
    • Search for and install:
    • Arduino_LSM9DS1
    • ArduinoBLE
    • Arduino_TensorFlowLite Manage libraries TensorFlow Lite Library
  2. Open the sketch and flash
    • Download the latest relase of tf4micro motion kit here
    • Open the arduino/tf4micro-motion-kit and double click on file
    • Click the Right arrow in the top left corner to build and upload the sketch.
      Arduino Port
  1. Go to the URL related to the experiment. The URL can be found below and play!

Using the TensorFlow Microcontroller Challenge Kit by SparkFun

The board that comes with the TensorFlow Microcontroller Challenge Kit by SparkFun comes preflashed with a sketch that will work with some of the experiments right out of the box. If you are using one of the “TensorFlow Micro” kits and you just want to jump right into playing with the experiments then you can simply connect your arduino to a power source (USB or Battery) and connect to one of the following experiment URLs:

FAQ & Common Errors

What exactly is being transferred when I “connect”? When you’re connecting the board to your computer, a pre-trained TensorFlow Lite machine learning model gets transferred over BLE onto the device. The sketches that are uploaded to the Arduino include a common TensorFlow Lite for Microcontrollers Experiments model architecture. The different experiment websites change the behavior of the sketch by changing the model to one specifically made for the experience.

What if I’m having issues connecting via bluetooth? If you are having issues connecting try the following:

  1. Make sure your browser (Chrome or Edge) supports Bluetooth and it is enabled.
  2. Make sure your device (laptop, phone, etc) supports Bluetooth and that it is working and enabled.
  3. Refresh the web page, unplug the Arduino power cable and then plug it back in to reset, then try connecting again.

NOTE: If you’re using a managed device, like a computer from school or work, your device policy may prevent BLE pairing.

My board isn’t showing up on my computer, even though it’s plugged in. What should I do? Try unplugging the Arduino power cable and then plug it back in to reset. Make sure you see the RGB LED blink red, green, blue in a sequential order.

The model isn’t getting my movements right. What do I do? The way you move may be different from the data we used to pre-train the model. Different people move differently. That’s why we created Tiny Motion Trainer, which lets you train a custom model based on the way you move.

Where should I go from here if I want to make my own model or project? You can create your own model in several different ways. Check out these links:

“What sensors do the experiments use?”
The IMU is a LSM9DS1. It is a 3-axis accelerometer, 3-axis gyroscope and 3-axis magnetometer. This chip, made by ST Microelectronics, is a standard component supported by our library ArduinoLSM9DS1. Read more here: https://www.arduino.cc/en/Guide/NANO33BLESense

How do you shrink a TensorFlow model to fit on a microcontroller?
Post-training quantization is a conversion technique that can reduce model size while also improving CPU and hardware accelerator latency, with little degradation in model accuracy. You can quantize an already-trained float TensorFlow model when you convert it to TensorFlow Lite format using the TensorFlow Lite Converter. Read more here: https://www.tensorflow.org/lite/performance/post_training_quantization

ML Model

Air pencil is built with Tiny Motion Trainer, it lets you train and test IMU based TFLite models in the browser.

Initial

Initial Initial

Arduino Initial

Capture1.png Result.png
Gesture / : A forward slant gesture training Captured dataset
Train.png Test.png
Train the model Test using the same setup

Installation

Air Pencil requires Node.js v10+ to run. You need live-server-https, python2 or 3 installed

cd Edu-Pencil
npm install
sh serve.sh

Then go to https://localhost:8181 in your browser and follow instructions

License

MIT

This site is open source. Improve this page.