Rusty-machine
James Lucas
This talk
- What is machine learning?
- How does rusty-machine work?
- Why is rusty-machine great?
What is rusty-machine?
Rusty-machine is a machine learning library written entirely in Rust.
It focuses on the following:
- Works out-of-the-box without relying on external dependencies.
- Simple and easy to understand API.
- Extendible and easy to configure.
Another machine learning library?
Machine Learning
"Field of study that gives computers the ability to learn without being explicitly programmed." - Arthur Samuel
How do machines learn?
With data.
Some examples
- Predicting rent increase
- Predicting whether an image contains a cat or a dog
- Understanding hand written digits
Data set might be:
rent prices and other facts about the residence.
labelled pictures of cats and dogs.
many examples of hand written digits.
Some terminology
- Model : An object that transforms inputs into outputs based on information in data.
- Train/Fit : Teaching a model how it should transform inputs using data.
- Predict : Feeding inputs into a model to receive outputs.
To predict rent increases we may use a Linear Regression Model. We'd train the model on some rent prices and facts about the residence. Then we'd predict the rent of unlisted places.
Why is machine learning hard?
There are many, many models to choose from.
There are many, many ways to use each model.
Back to rusty-machine
The foundation of rusty-machine
pub trait Model<T, U> {
fn train(&mut self, inputs: &T, targets: &U);
fn predict(&self, inputs: &T) -> U;
}
An example
Before we go any further we should see an example.
K-Means
A model for clustering.
Using a K-Means Model
// ... Get the data samples
// Create a new model with 2 clusters
let mut model = KMeansClassifier::new(2);
// Train the model
model.train(&samples);
// Predict which cluster each point belongs to
let clusters : Vector<usize> = model.predict(&samples);
You can run the full example in the rusty-machine repo.
Under the hood
K-Means works in roughly the following way:
- Get some initial guesses for the centroids (cluster centers)
- Assign each point to the centroid it is closest to.
- Update the centroids by taking the average of all points assigned to it.
- Repeat 2 and 3 until convergence.
K-Means Classification
Simple but complicated
The API for other models aim to be as simple as that one. However...
Rusty-machine aims for ease of use.
How does rusty-machine (try to) keep things simple?
Using traits
- A clean, simple model API
- Extensibility at the user level
- Reusable components within the library
Extensibility
We use traits to define parts of the models.
While rusty-machine provides common defaults - users can write their own implementations and plug them in.
Extensibility Example
Support Vector Machine
/// A Support Vector Machine
pub struct SVM<K: Kernel> {
ker: K,
/// Some other fields
/* ... */
}
pub trait Kernel {
/// The kernel function.
///
/// Takes two equal length slices and returns a scalar.
fn kernel(&self, x1: &[f64], x2: &[f64]) -> f64;
}
Combining kernels
K1(x1, x2) + K2(x1, x2) = K(x1, x2)
pub struct KernelSum<T, U>
where T: Kernel,
U: Kernel
{
k1: T,
k2: U,
}
/// Computes the sum of the two associated kernels.
impl<T, U> Kernel for KernelSum<T, U>
where T: Kernel,
U: Kernel
{
fn kernel(&self, x1: &[f64], x2: &[f64]) -> f64 {
self.k1.kernel(x1, x2) + self.k2.kernel(x1, x2)
}
}
Combining kernels
K1(x1, x2) + K2(x1, x2) = K(x1, x2)
let poly_ker = kernel::Polynomial::new(...);
let hypert_ker = kernel::HyperTan::new(...);
let sum_kernel = poly_ker + hypert_ker;
let mut model = SVM::new(sum_kernel);
Reusability
We use traits to define common components, e.g. Kernels.
These components can be swapped in and out of models.
New models can easily make use of these common components.
Reusability Example
Gradient Descent Solvers
We use Gradient Descent to minimize a cost function.
All Gradient Descent Solvers implement this trait.
/// Trait for gradient descent algorithms. (Some things omitted)
pub trait OptimAlgorithm<M: Optimizable> {
/// Return the optimized parameters using gradient optimization.
fn optimize(&self, model: &M, ...) -> Vec<f64>;
}
The Optimizable trait is implemented by a model which is differentiable.
Creating a new model
With gradient descent optimization
Define the model.
/// Cost function is: f(x) = (x-c)^2
struct XSqModel {
c: f64,
}
You can think of this model as learning the value c.
Creating a new model
With gradient descent optimization
Implement Optimizable for model.
/// Cost function is: f(x) = (x-c)^2
struct XSqModel {
c: f64,
}
impl Optimizable for XSqModel {
/// 'params' here is 'x'
fn compute_grad(&self, params: &[f64], ...) -> Vec<f64> {
vec![2f64 * (params[0] - self.c)]
}
}
Creating a new model
With gradient descent optimization
Use an OptimAlgorithm to compute the optimized parameters.
/// Cost function is: f(x) = (x-c)^2
struct XSqModel {
c: f64,
}
impl Optimizable for XSqModel {
fn compute_grad(&self, params: &[f64], ...) -> Vec<f64> {
vec![2f64 * (params[0] - self.c)]
}
}
let x_sq = XSqModel { c : 1.0 };
let x_start = vec![30.0];
let gd = GradientDesc::default();
let optimal = gd.optimize(&x_sq, &x_start, ...);
What can rusty-machine do?
- K-Means Clustering
- DBSCAN Clustering
- Linear Regression
- Logistic Regression
- Generalized Linear Models
- Neural Networks
- Gaussian Process Regression
- Support Vector Machines
- Gaussian Mixture Models
- Naive Bayes Classifiers
Linear Algebra - Rulinalg
Rusty-machine works without any external dependencies.
Rulinalg provides linear algebra implemented entirely in Rust.
Why Rulinalg?
Ease of use
A quick note on error handling
Rust's error handling is fantastic.
impl Matrix<T> {
pub fn inverse(&self) -> Result<Matrix<T>, Error> {
// Fun stuff goes here
}
}
What does Rulinalg do?
- Data structures (
Matrix,Vector) - Basic operators (with in-place allocation where possible)
- Decompositions (Inverse, Eigendecomp, SVD, etc.)
- And more...
Why is Rust a good choice?
- Trait system is amazing.
- Error handling is amazing.
- Performance focused code*.
* Rusty-machine needs some work, but the future looks bright!
Why is Rust a good choice?
Most importantly for me - safe control over memory.
When would you use rusty-machine?
At the moment - experimentation, non-performance critical applications.
In the future - quick, safe and powerful modeling.
Rust and ML in general
What's next?
- Optimizing and stabilizing existing models.
- Providing optional use of BLAS/LAPACK/CUDA/etc.
- Addressing lack of tooling.
What would I like to see from Rust?
- Specialization
- Growth of Float/Complex generics
- Continued effort from community
Summary
- Machine learning (done quickly)
- Rusty-machine
- Rulinalg
Contributors
| zackmdavis | DarkDrek | tafia | |
| ic | rrichardson | vishalsodani | |
| raulsi | danlrobertson | brendan-rius | |
| andrewcsmith |
Thanks!
Some Links
Some FAQs
Why no GPU support
From Scikit-learn's FAQs.
BLAS/LAPACK
Hopefully soon!
Integrating with other languages
Nothing planned yet, but some good choices.
Python is especially exciting as we gain access to lots of tooling.