Realtime perceptual and interaction capabilities in mixed reality require a range of 3D tracking problems to be solved at low latency on resource-constrained hardware such as head-mounted devices. Indeed, for devices such as HoloLens 2 where the CPU and GPU are left available for applications, multiple tracking subsystems are required to run on a continuous, real-time basis while sharing a single Digital Signal Processor. To solve model-fitting problems for HoloLens 2 hand tracking, where the computational budget is approximately 100 times smaller than an iPhone 7, we introduce a new surface model: the 'Phong surface'. Using ideas from computer graphics, the Phong surface describes the same 3D shape as a triangulated mesh model, but with continuous surface normals which enable the use of lifting-based optimization, providing significant efficiency gains over ICP-based methods. We show that Phong surfaces retain the convergence benefits of smoother surface models, while triangle meshes do not.

@inproceedings{Shen:2020:TPS,
  title   = {The Phong Surface: Efficient 3D Model Fitting using
             Lifted Optimization},
  author  = {Jingjing Shen and Thomas J. Cashman and Qi Ye and
             Tim Hutton and Toby Sharp and Federica Bogo and Andrew
             Fitzgibbon and Jamie Shotton},
  booktitle = {Proceedings of the European Conference on Computer
               Vision (ECCV)},
  year    = {2020}
}

Fully articulated hand tracking promises to enable fundamentally new interactions with virtual and augmented worlds, but the limited accuracy and efficiency of current systems has prevented widespread adoption. Today's dominant paradigm uses machine learning for initialization and recovery followed by iterative model-fitting optimization to achieve a detailed pose fit. We follow this paradigm, but make several changes to the model-fitting, namely using: (1) a more discriminative objective function; (2) a smooth-surface model that provides gradients for non-linear optimization; and (3) joint optimization over both the model pose and the correspondences between observed data points and the model surface. While each of these changes may actually increase the cost per fitting iteration, we find a compensating decrease in the number of iterations. Further, the wide basin of convergence means that fewer starting points are needed for successful model fitting. Our system runs in real-time on CPU only, which frees up the commonly over-burdened GPU for experience designers. The hand tracker is efficient enough to run on low-power devices such as tablets. We can track up to several meters from the camera to provide a large working volume for interaction, even using the noisy data from current-generation depth cameras. Quantitative assessments on standard datasets show that the new approach exceeds the state of the art in accuracy. Qualitative results take the form of live recordings of a range of interactive experiences enabled by this new approach.

@article{Taylor:2016:EPI,
  title   = {Efficient and Precise Interactive Hand Tracking
             through Joint, Continuous Optimization of Pose and
             Correspondences},
  author  = {Jonathan Taylor and Lucas Bordeaux and Thomas Cashman
             and Bob Corish and Cem Keskin and Eduardo Soto and
             David Sweeney and Julien Valentin and Benjamin Luff
             and Arran Topalian and Erroll Wood and Sameh Khamis
             and Pushmeet Kohli and Toby Sharp and Shahram Izadi
             and Richard Banks and Andrew Fitzgibbon and
             Jamie Shotton},
  journal = {ACM Transactions on Graphics},
  year    = {2016},
  volume  = {35},
  number  = {4},
  pages   = {\#143, 1--12}
}

3D morphable models are low-dimensional parametrizations of 3D object classes which provide a powerful means of associating 3D geometry to 2D images. However, morphable models are currently generated from 3D scans, so for general object classes such as animals they are economically and practically infeasible. We show that, given a small amount of user interaction (little more than that required to build a conventional morphable model), there is enough information in a collection of 2D pictures of certain object classes to generate a full 3D morphable model, even in the absence of surface texture. The key restriction is that the object class should not be strongly articulated, and that a very rough rigid model should be provided as an initial estimate of the 'mean shape'.

The model representation is a linear combination of subdivision surfaces, which we fit to image silhouettes and any identifiable key points using a novel combined continuous-discrete optimization strategy. Results are demonstrated on several natural object classes, and show that models of rather high quality can be obtained from this limited information.

Full MATLAB source code is available from CodePlex.

The CodePlex release also contains our data sets for bananas, pigeons, polar bears and (of course) dolphins.

See the included documentation to reproduce our results.

@article{Cashman:2013:WSD,
  author  = {Thomas J. Cashman and Andrew W. Fitzgibbon},
  title   = {What shape are dolphins? Building {3D} morphable
             models from {2D} images},
  journal = {IEEE Transactions on Pattern Analysis and Machine
             Intelligence},
  volume  = 35,
  number  = 1,
  pages   = {232--244},
  year    = 2013
}

It is increasingly popular to represent non-rigid motion using a deforming mesh sequence: a discrete sequence of frames, each of which is given as a mesh with a common graph structure. Such sequences have the flexibility to represent a wide range of mesh deformations used in practice, but they are also highly redundant, expensive to store, and difficult to edit in a time-coherent manner. We address these limitations with a continuous representation that extracts redundancy in three separate phases, leading to separate editable signals in time, pose and shape. The representation can be applied to any deforming mesh sequence, in contrast to previous domain-specific approaches. By modifying the three signal components, we demonstrate time-coherent editing operations such as local repetition of part of a sequence, frame rate conversion and deformation transfer. We also show that our representation makes it possible to design new deforming sequences simply by sketching a curve in a 2D pose space.

Binaries for Windows (32 bit, 8.2 MB)
We use CHOLMOD for solving sparse linear systems in certain shape spaces. CHOLMOD can be compiled with the graph partitioning software METIS to give better performance, but can not be distributed in this form under the terms of the GNU GPL. The version of CHOLMOD in this implementation has therefore been compiled without METIS, but you may gain better performance by compiling, from source, against CHOLMOD linked with METIS and/or a version of the BLAS which has been optimized for your processor.

Source code (97 KB)

Sample 'flying squirrel' mesh sequence from Big Buck Bunny (8.3 MB)

Sample 'flag' mesh sequence (36.5 MB)

@article{Cashman:2012:CER,
  author  = {Thomas J. Cashman and Kai Hormann},
  title   = {A continuous, editable representation for deforming
             mesh sequences with separate signals for time, pose
             and shape},
  journal = {Computer Graphics Forum},
  volume  = 31,
  number  = 2,
  year    = 2012,
  pages   = {735--744},
  note    = {Proceedings of Eurographics}
}

We present a subdivision framework that adds extraordinary vertices to NURBS of arbitrarily high degree. The surfaces can represent any odd degree NURBS patch exactly. Our rules handle non-uniform knot vectors, and are not restricted to midpoint knot insertion. In the absence of multiple knots at extraordinary points, the limit surfaces have bounded curvature.

Binaries for Windows (32 bit, 5.4 MB)

Source code (960.4 KB)

Precomputed tables containing bounded curvature solutions (68.6 KB)

@article{Cashman:2009:NEP,
  title   = {{NURBS} with Extraordinary Points:
             High-degree, Non-uniform, Rational Subdivision
             Schemes},
  author  = {Thomas J. Cashman and Ursula H. Augsd{\"o}rfer and
             Neil A. Dodgson and Malcolm A. Sabin},
  journal = {ACM Transactions on Graphics},
  year    = {2009},
  volume  = {28},
  number  = {3},
  pages   = {\#46, 1--9}
}