The scope of LEARNAS is to her people who are threatened by disabilities in learning and daily live. Examples for such people are slow learning children, or people with language barriers, such as refugees.
History and Background
Since a long time, the Pestalozzischule in Heilbronn, Germany, is leading efforts to address the special needs of slow learning children in education and training. This has led to cooperation with industry and the city of Heilbronn. The project is called "Training on the Job" and aims to train students on actions that they need in daily live or potential future jobs. In this project Pestalozzischule trains students in professional textile cleaning or service. In this scope Pestalozzischule runs a coffee shop that is ran by students and teachers. This has lead to supra-regional reputation of the school.
The speciality of the concept is that children are not only trained theoretically, as traditional schools do, but are also trained specifically for certain jobs to allow them to gain practical knowledge straight from school. This concept has been proven to be extremely successful. A significant amount of students who successfully leave Pestalozzischule can get into jobs that require a high amount of knowledge although these students are weaker in learning. Due to the concept of "Training on the job", multiple employers provide opportunities to those well trained students. Another significant amount of those students is able to continue school lower secondary education.
A core principle of the concept is close support and assistance. There are personal learning plans for students and there is a high work effort of teachers, parents and supporters. Several open questions need however to be addressed:
- How can students learn at home?
- How can one measure learning success?
- What if students do not want to ask because they feel ashamed of asking?
- What if students suffer from disabilities that make it hard to interact with people, such as autism or talking or hearing disabilities?
- How can the learned be trained to remember? There is a need for repetition and repeatability.
- How can the concept be scale? 1:1 assistance is not possible due to the high workload for teachers.
This is where LEARNAS can help and where it originated from. Although having high potential for personal learning solutions in schools, it can also assist in other areas, which is shown in the outlook.
Concept and Aim
LEARNAS stems from the need of personal learning assistance. The novelty is the interdisciplinary approach and the close work with the Pestalozzischule and its' network. Requirements will be specified throughout the development process and will be refined in a prototypic manner. At the time being, there is no more than an idea and a concept, which is described in the following.
The Software developed in this scope will target learning problems that are being faced right now in the Pestalozzischule. The first problem that is being targeted is "the cooking and baking problem".
Pestalozzischule is running a kitchen for serving food to their students that is run by students and teachers to assist the students. The kitchen serves food for the students but is also a bakery for the coffee shop of the Pestalozzischule.
An exemplaric problem is the baking of a cake, which can be problematic for students. Imagine you are baking a cake. What we want is clear but how can we tackle this problem? We need to follow specific steps that are essential to learning and to the solution finding process.
- Find and choose a recipe
- Find the ingredients
- Find the necessary equipment.
- Prepare the dough step by step.
- Prepare the cake out of dough and ingredients.
- Handle the oven and bake the cake.
- Handle the cake after baking.
- Cleaning of the kitchen.
All this is in the recipe and if you did that once you probably can do it a second time. It may even sound trivial to some of the readers. However, this does not apply to a huge amount of weak learners and some of them might be afraid of asking or have problems in communicating. Each of the steps can be a challenge for them.
How can we help with software based assistance? There are several options that are shown in the following:
- Augmented reality
- Measuring success
Step 1: Checklist
We will start to develop a software that can work with checklists and provide the necessary assistance to students who can read and understand what they have read. These lists will be customized to the needs of the Pestalozzischule. The recipes and equipment as well as the ingredients used by the Pestalozzischule will be configured into the checklist and students will have a full manual on what they bake, how they do it and where they can find the ingredients and equipment in the kitchen of Pestalozzischule. This will be achieved by textual as well as image based check lists.
Step 2: Augmented reality: Basic functionality
We will leverage camera technologies to improve the system and to make interaction more natural and accessible for student that have language barriers or who struggle with reading or abstraction. In order to achieve this, the integrated camera of the mobile devices as well as the speaker of the device can be used. The student will be able to point the device on any of the equipment (after it has been learned by the software) and the device will respond graphically or by the output of voice. This will allow students to understand what is seen and what they can and have to do with the objects. The software will also guide through the workflow. (gleichzeitig werden die sprachlichen Fähigkeiten erweitert/Wortschatztraining)
If the teacher assists the student by placing all the necessary ingredients and equipment in the working area, a student can use the device to bake the cake by pointing the device on the working area and by following the instructions. The software will lead the student through the process by giving precise instructions what has to be done.
Step 3: Augmented reality: Mobile devices might be inconvenient
Imagine you are baking a cake. You cannot hold a mobile phone or a tablet in your hand throughout the whole process if you want to be efficient. We should bring the concept to augmented reality devices such as the Microsoft HoloLens or other handsfree augmented reality devices/glasses.
Step 4: Augmented reality: Localization
It is not a natural scenario that all the equipment is placed within the working area. It is not even natural that everything is available and assistance in the supermarket may be required. For now we assume that everything is already in the kitchen. You need to find the equipment. We will create a map of the kitchen by using SLAM methods (Simultaneous Localization and Mapping) or by using external 3D sensors such as the Microsoft Kinect camera. We will then use this map and the sensors to localize the devices and the students in the kitchen to guide them to the locations where they can find equipment in order to complete the process if no assistance by a teacher is provided.
Step 5: Measuring success
With the technological foundation we have everything to measure success. We can give full assistance to students at the beginning of the learning process and reduce the level of assistance after several repetitions (wenn klar ist, dass der trainierte Bereich gelernt ist und keiner Hilfe mehr bedarf). We can also measure durations and compare them to previous run-throughs in order to quantitatively measure the learning process.
Technological foundations and Contributions
Why is a robotics engineer interested in learning technologies? On the one hand i am closely connected (the principal is a relative ;-) ) to this school and have followed the actions and the passion of the involved teachers for 20 years. On the other hand you might already have noticed that the technology being used mostly originated from robotics.
How to address the challenges from a software point of view?
There are so many open questions about devices, platforms, and usability, so that a cross platform toolkit with a high grade of customization capabilities is a natural choice. Being a C++ developer, Qt is a natural choice. It comes with a rich toolchain, a lot of features, runs everywhere and the GUI is extremely customizable. However, commercial exploitation may require a costly developer license. Wait? You said commercial exploitation? This is Hackaday and this is about sharing! Yes it is. We are going to talk about this in the licensing chapter and we will share our work!
QT is capable of building a complete application for mobile devices that will allow to achieve the first step of the project and that will be able to serve as a first in field prototype.
As you might have noticed, this is not a short term project and therefore a solid foundation is required. Technically we can continue from there using widespread robotics and imaging frameworks:
The 2D image processing will be done in OpenCV, while 3D Pointcloud processing will be performed in PCL. In order to distribute data and to modularize, ROS will be used as a common foundation of the whole system. This allows to leverage additional computing power if the built in processor of mobile devices cannot handle the processing and it allows to comfortably connect external sensors such as the cameras to the system.
Some things such as the machine learning aspect (object detection and Workflow tracking) are not yet known so i have not decided yet which learning framework will be used. I will probably look into OpenCV, SciKit Learn, Tensorflow or Theano.
- Build a first prototype for the checklist based software. Perform first experiments with and in the Pestalozzischule.
- Put out a first shot on image processing and object detection
- Found a company that is dedicated to the development of assistance systems.
2018 - Open End:
- All other of the steps shown in "Concept and Aim" and wherever the journey will lead
The concept is new and has not been tried before. There are multiple significant risks.
- Firstly, the software may not efficiently assist in the learning process.
- Moreover, object detection and situation understanding can be incredibly hard to implement.
- The system has to be self-learning. This is even harder.
- The project is huge and I am a full-time worker in a company and can only dedicate a limited amount of my time to this project.
- We are working with people! This introduces non technical aspects and an interactive character to the project which consumes a lot of time.
This project is something new and has so much potential for any assistance technology that is process based. It would be awesome to put out a framework that can help with learning and assistance in day to day situations for everyone and that can be context sensitive and aware of its users!
Licensing, Accessibility and Organization
LEARNAS is a project that is primarily driven by myself. I am a robotics engineer and there is no big team behind this.
LEARNAS however has been developed from a cooperation with Pestalozzischule Heilbronn that has incredible experience in alternative learning methods. One of the most important aspects is that school has to be adapted to the child and not the other way round.
Our primary goal is to deliver support for the children that are studying at Pestalozzischule. These children are slow learners and/or suffer from diseases such as autism. LEARNAS shall be developed together with the Pestalozzischule and shall get into daily use of the Pestalozzischule.
The scope, however, is wider! Pestalozzischule always strived to open their methods to other institutions and schools and is in close contact with them to provide the methodology. Every child who has similar needs shall be able to profit from the methodology.
LEARNAS will follow the same principles. Pestalozzischule will be a pilot for LEARNAS, which will hopefully be distributed to other institutions, schools, refugee centers or employment offices. Eventually essences of it might even end up as an application for everyone.
Why don't we want to open everything? Why are we looking for commercial exploitation and which parts of the project will be open and which will be closed?
- LEARNAS is personalized. It fits the needs of Pestalozzischule and might not be applicable to other areas in the beginning. This renders the software unusable for other institutions. Additionally, a lot of expertise is required by the assistants (i.e. the teachers) who assist students to use the software. Uncontrolled distribution of the software might lead to the impression that the concept is not working or that thirds are trying to commercialize the consulting aspect of the software. Bad publicity is the worst for a project that tries to alter established methods that stick to peoples minds.
- Distribution of such a project is expensive. We need to do networking, we need to roll things out and we are not a big team. Portions of the project need to be closed to secure IP and to gain funding for distribution and for widening the scope of the project. If things run well we need to dedicate additional resources. These things do not come for free. To put that onto a solid base I will found a company that is dedicated to the development of LEARNAS and that gives the project a more official shape.
- This is and shall be a community project. We do want to get feedback and we do want to give things back. Although the direct user interface and large portions of the application will be closed source, we are willing to share. We do want to showcase a first prototype of the checklist-based software that we will open up under a BSD, GPL oder MIT style license so that Hackaday readers can see what we are doing. Additionally we will document our framework, our methodology, how we develop and how we interface engineering and pedagogics. We will also show the results of our experiments and we will ask for feedback if it is needed. We will showcase the architecture and we will share large portions of the robotics framework for everything beyond the scope of step one (It is ROS-based, so there are already a lot of things that can be leveraged and that are free software).
- There is absolutely no intention to charge people in need for LEARNAS and we will not charge students, parents or the Pestalozzischule for the technology. Instead we are trying to push out the methodology (in case it works) to as many people as possible! We want to create something that lasts!
- We are not completely on our own and we need permission by the city of Heilbronn before we start to go into Pestalozzischule. We are working on this, but there might be restrictions that need to be addressed.
Future and Outlook
The software is intended to help slow learning children. But we want to learn as well. We want to make learning methodology graspable, customizable and modern.
The idea is to provide a learning tool for everyday. This cannot only help students to bake a cake or to gain home working skills, it can also help to assist refugees to do their day to day work while being new to the language (e.g. getting directions and manuals using visual aids only, without the need of reading). Being led through a process without the need of speaking one and the same language (don't get it wrong. We want people in a country to speak the same language. But if someone is new, it can be hard) can help to find one's way into a new culture and society. We are convinced this can help to overcome language barriers, can make learning more efficient and maybe it can also be of use for everybody. You might be able to learn e.g. wood or metalworking or other things with technology like this.
Think big! Maybe there can also be something like a caring robot that helps people to learn, that guides or assists with certain.
duties and that is equipped with this technology!