System Model

Architecture of MyPath Project.

Due to the wide variation in user-wheelchair-environment combinations, it is simply impossible to collect data from every different type of user on all possible types of wheelchairs covering all the different surfaces of the world. Therefore, we applied a two-pronged approach to solve this problem. We use a sensor-based data collection and path classification module for training the system on certain amounts of data. We use that data and knowledge to deduce information about other types of users, wheelchairs and surfaces. This project gives a very basic idea of knowledge transfer by the use of transfer learning techniques. Also, in response to the darth of a significant amount of sensor generated surface data, we have employed an image based surface classification system in this work. We collected surface images using a car-mounted Go-Pro camera and classified surfaces using a large set of images which we used to train another machine learning model. Machine learning is a method of data analysis that allows you to learn data iteratively and analyze models. Machine learning plays a big role in MyPath project because we need to analyze the wheelchair travel of a large number of users. When the model encounters new data, we need to use iterations to guide the machine to adapt to the new data independently.

MyPath - The Bigger Picture

MyPath - The Bigger Picture.

In order to provide individuals with restricted mobility the opportunity to travel more efficiently, various systems have proposed modeling techniques and routing algorithms that handle accessible navigation through the built environment which is otherwise dotted with mobility barriers. Such systems use data gathered from smartphone sensors or crowd-sourcing to pinpoint the location of the barriers as well as the facilities, such as crosswalks with traffic signals or access ramps to curbs. Though the previous works have identified the type of surface and incline to be important features to determine accessibility, no extensive empirical research exists on how these parameters affect navigation. In order to address this problem, we propose to build a novel system called WheelShare, which uses machine learning to classify surfaces into accessible or otherwise and uses that knowledge to generate accessible routes for wheelchair users. We have trained our system with accelerometer and gyroscope data obtained from 26 different surfaces found frequently in indoor and outdoor environments across Europe and USA. More data is collected by the system through crowd-sourcing based contribution from interested users. Our evaluation shows that WheelShare can achieve an accuracy of up to 96% in identifying surfaces in one of the 5 different accessibility classes. Overall, WheelShare is a novel, scalable and data-centric approach to objectively identify the accessible features of a surface and can generate end-to-end routes for wheelchair users using frequently updated crowd-sourced information.

Target Audience

The project’s target audience is people with disabilities which affect their physical movement (i.e. people who use wheelchairs, crutches, etc). The app, however, can be used by anyone in the general public since it will help the user to navigate from point A to point B by giving an accessibl route to them.

Stand-out Features

The route is determined by an algorithm that takes the user’s personal physical information as well as their ASIA (American Spinal Injury Association) Impairment Scale score into account. Users contribute directly to the algorithm by contributing information such as geolocation and sensors.

Technical Know

A mobile app for the project has been created using Xamarin, C#, and Visual Studio.

Client Side Working of the App

The user enters source and destination location coordinates to initiate a route request. The server returns multiple accessible routes with different path features which are then personalized based on user preferences such as acceptable curb heights, or slopes. Also, a textual navigation pane is provided for the chosen route which gives turn-by-turn navigational instructions with distance, time and surface type information to cover each section of the route.

Machine Learning

We have used sensor data to train the ML model on the road surfaces which users access with a wheelchair. In the Machine Learning code, we use 80% supervised learning and 20% unsupervised learning for all models. We have decided to use Random Forests(RF) and Artificial Neural Networks(ANN) as the final Machine Learning Models. Although deep learning is more expensive, we have still chosen deep learning because machine learning provides only a simple level of feature extraction, and some information with discrimination cannot be extracted effectively. Some models need to be artificially designed or applied to certain features as input to the Machine Learning model. After model decision, we have used Transfer Learning to transfer manual wheelchair dataset to power wheelchair dataset. The reason we used transfer learning is that there is a significant difference between the two datasets. Here are the various models we used:

  • Machine Learning Models
    • Logistic Regression (LR)
    • Linear Discriminant Analysis (LDA)
    • Decision Trees (DT)
    • K-Nearest Neighbors (KNN)
    • Gaussian Naïve Bayes (NB)
    • Random Forests (RF)
    • Support Vector Machines (SVM)
  • Deep Learning Models
    • Artificial Neural Network (ANN)
    • Convolutional Neural Network (CNN)
    • Long Short-Term Memory Networks (LSTMs)
  • Transfer Learning Models
    • Adaboost Algorithm based on fixed layers of ANN model
    • Tradaboost Algorithm based on ANN model

Idea of Image Data

Sometimes the users are not able to access the wheelchair, thus we have implemented another approach to evaluate accessibility by using images collected from many different accessible surfaces to train a computer vision based model.

Labeling

Using LabelMe library in Jupyter Notebook to identify each surface type and barriers in the image.

Image Processing

We have decided to build up the model base on Deeplabv3 and Resnet101. First, we prepare the images and mask data and set them up for training, then we run the training process of the model and finally we evaluate the predictions and model performances. After that we are able to cut down the loss to around 0.3 level and we are able to add this new feature to the project.

Mainstream Methodologies for Image Classification:

  • Pure Image Classification
  • Pixel-level Segmentation
  • Object Detection Based Methods

Crack Detection Methodologies:

  • Binary Patterns
  • Gabor Filter
  • Shape Based Methods
  • Tree Structure Algorithms

Routing and Graphhoper

Process of calculating the navigation path:

  • The Front-end client sends a query to the server, including the coordinates of start, endpoint, and other information.
  • Calculate accessibility scores based on various road information and user information.
    1. The road info including the surface type and the slope of the road.
    2. The user info including the age, height, weight.
  • Assign weights to roads based on the calculated accessibility scores.
  • Generate 2 routes based on those assigned scores.
  • Import and modify the Graphhopper API.
  • Send outcome results back to the front-end client.