FluentRealSense – the first steps to a simpler RealSense

Those who know me are aware that I have a long term association (nay, let’s say it for what it is, love affair) with the RealSense platforms from Intel. If you’ve been following developments in that space for any length of time, you’re probably aware that things have moved on a lot and the cameras are available in all sorts of devices and are no longer just limited to the Windows platform. This shift of emphasis means that Intel have moved away from concentrating on the old monolithic RealSense SDK and have moved to supporting a new open source API (available here: https://github.com/IntelRealSense/librealsense).

I have spent a period of time working with this API as it has evolved and have started “wrapping” it to make it a little bit friendlier to use (in my opinion) for people who don’t want to get bogged down with the “nitty gritty” of how to get devices, and so on. This work has been in C++, the language the API is available in, and I am going to cover, over a series of posts, how this library evolved. I’ll show you the code I’ve written at each step so that you can also see how I have gone about relearning modern C++ (bearing in mind it has been 18 years since I last worked with it in anger) so you will definitely find that early iterations of the code are naive, at best. My aim with this library was to make it as fluent as possible, hence it’s name, FluentRealSense.

An upfront note. Early iterations of this codebase have been done entirely on Windows running Visual Studio 2017. I chose this particular environment because, a) Visual Studio is, to my mind, the finest IDE around and b) because 2017 supports porting your C++ code directly over to Linux. This lets me leverage the rapid turnaround time I’m used to in my dev environment and the ease of deploying to my intended targets later on. Thank you Microsoft for keeping this old developer happy.

Let’s start with the first iteration. The first thing I wanted the code to do was to provide me with the ability to iterate over all the cameras on my system and print out some diagnostic information. This meant that my code would be broken down initially into three classes:
information: This class reads information from the API about a single camera and returns it as a string.
camera: This is a single device. Initially, it’s just going to instantiate and provide access to the information class.
cameras: The entry point for consuming applications, this class is responsible for finding out about and managing access to each camera; this class is also enumerable so that the calling code can quickly iterate over each camera.

Let’s start off by looking at the information class. First we’re going to add some includes and namespaces:

#pragma once
#include "hpp/rs_context.hpp"

using namespace std;
using namespace rs2;

class information
{
public:
class information() {}
~information() {}
}

That’s all very straightforward and nothing you won’t have seen before. Let’s start fleshing this out.

The first thing we’re going to need is a RealSense device to get information from – this is going to be passed in so let’s add a member to store it and replace our constructor with this:

explicit information(const device device) : device_(device) {}
:::
private:
device device_;

At this stage, our code looks like this:

#pragma once
#include "hpp/rs_context.hpp"

using namespace std;
using namespace rs2;

class information
{
public:
explicit information(const device device) : device_(device) {}
~information() {}

private:
device device_;
}

I have to say, at this point, I love the improvements to instantiating members that C++ now provides. This is a wonderful little innovation.

Now, in order to get the information out of the API, I’m going to add a handy little private helper method. This makes use of the fact that the API exposes this information via an rs2_camera_info enumeration:

const char* get_info(const rs2_camera_info info) const
{
if (!device_.supports(info))
{
return "Not supported";
}
return device_.get_info(info);
}

This is the first point that our code is going to add any value. Whenever we call get_info against the underlying device, we have to check that that particular type of info can be retrieved for that device. It’s very easy to write code that makes assumptions that each camera supports exactly the same set of properties here so the underlying call could throw an exception. To get around this, our code checks to see if the device supports that particular rs2_camera_info type and returns “Not supported” if it doesn’t. It is so easy to forget that you should always pair the get_info with the supports, so this helper method removes the need to remember that.

So, how do we use this? Well, with some public methods like these:

const char* name() const
{
return get_info(RS2_CAMERA_INFO_NAME);
}

const char* serial_number() const
{
return get_info(RS2_CAMERA_INFO_SERIAL_NUMBER);
}

const char* port() const
{
return get_info(RS2_CAMERA_INFO_PHYSICAL_PORT);
}

const char* firmware_version() const
{
return get_info(RS2_CAMERA_INFO_FIRMWARE_VERSION);
}

const char* debug_opCode() const
{
return get_info(RS2_CAMERA_INFO_DEBUG_OP_CODE);
}

const char* advanced_mode() const
{
return get_info(RS2_CAMERA_INFO_ADVANCED_MODE);
}

const char* product_id() const
{
return get_info(RS2_CAMERA_INFO_PRODUCT_ID);
}

const char* camera_locked() const
{
return get_info(RS2_CAMERA_INFO_CAMERA_LOCKED);
}

string dump_diagnostic() const
{
string text = "\nDevice Name: ";
text += name();
text += "\n Serial number: ";
text += serial_number();
text += "\n Port: ";
text += port();
text += "\n Firmware version: ";
text += firmware_version();
text += "\n Debug op code: ";
text += debug_opCode();
text += "\n Advanced Mode: ";
text += advanced_mode();
text += "\n Product id: ";
text += product_id();
text += "\n Camera locked: ";
text += camera_locked();

return text;
}

We now have our complete information class. It looks like this:

#pragma once
#include "hpp/rs_context.hpp"

using namespace std;
using namespace rs2;

class information
{
public:
explicit information(const device device) : device_(device) {}
~information()
{
}

const char* name() const
{
  return get_info(RS2_CAMERA_INFO_NAME);
}

const char* serial_number() const
{
  return get_info(RS2_CAMERA_INFO_SERIAL_NUMBER);
}

const char* port() const
{
  return get_info(RS2_CAMERA_INFO_PHYSICAL_PORT);
}

const char* firmware_version() const
{
  return get_info(RS2_CAMERA_INFO_FIRMWARE_VERSION);
}

const char* debug_opCode() const
{
  return get_info(RS2_CAMERA_INFO_DEBUG_OP_CODE);
}

const char* advanced_mode() const
{
  return get_info(RS2_CAMERA_INFO_ADVANCED_MODE);
}

const char* product_id() const
{
  return get_info(RS2_CAMERA_INFO_PRODUCT_ID);
}

const char* camera_locked() const
{
  return get_info(RS2_CAMERA_INFO_CAMERA_LOCKED);
}

string dump_diagnostic() const
{
  string text = "\nDevice Name: ";
  text += name();
  text += "\n Serial number: ";
  text += serial_number();
  text += "\n Port: ";
  text += port();
  text += "\n Firmware version: ";
  text += firmware_version();
  text += "\n Debug op code: ";
  text += debug_opCode();
  text += "\n Advanced Mode: ";
  text += advanced_mode();
  text += "\n Product id: ";
  text += product_id();
  text += "\n Camera locked: ";
  text += camera_locked();
  return text;
}

private:
  const char* get_info(const rs2_camera_info info) const
  {
    if (!device_.supports(info))
    {
      return "Not supported";
    }
    return device_.get_info(info);
  }
  device device_;
};

The next thing we have to do is write a class that represents a single RealSense camera. There’s not much to this class, at the moment, so let’s look at it in its entirety.

#pragma once
#include "hpp/rs_context.hpp"
#include "information.h"

using namespace std;
using namespace rs2;

class camera
{
public:
  explicit camera(const device dev) : information_(make_shared(dev)) {}

  ~camera()
  {
  }

  shared_ptr get_information() const
  {
    return information_;
  }

private:
  shared_ptr information_;
};

I did say this class was pretty light at the moment. The class accepts a single RealSense device which is used to instantiate the information class. We provide one method which is used to get the instance of the information class. That’s it so far.

Finally, we come to the entry point of our code, the cameras class. This class let’s us enumerate all of the cameras on our system and access the functions inside. As usual, we’ll start off with the definition:

#pragma once
#include 
#include 
#include "camera.h"
#include "hpp/rs_context.hpp"

using namespace std;
using namespace rs2;

class cameras
{
public:
  cameras() {}
  ~cameras() {}
}

As you will remember, I said that I wanted the cameras to be enumerable so I need to do some upfront declarations:

using cameras_t = vector<shared_ptr>;
using iterator = cameras_t::iterator;

using const_iterator = cameras_t::const_iterator;
[/source]

With these in place, I can now start to add the ability to enumerate over the camera instances. Before I do that, though, it’s time to introduce something new. In the preceding code, we saw that the RealSense camera was represented as a device that we passed into the relevant constructors. The question is, how did we get that device in the first place? Well, that’s down to the API providing a context that allows us to access these devices. So, let’s add a member to store a vector of camera instances and then build in the method to get the list of devices.

cameras() : cameras_(make_shared())
{
  context context;
  // Iterate over the devices;
  auto devices = context.query_devices();
  for (const auto dev : devices)
  {
    const auto cam = make_shared(dev);
    cameras_->push_back(cam);
  }
}

private:
  shared_ptr cameras_;

There’s nothing complicated in that code. We get the devices from the context using query_devices and iterate over each one, adding a new camera instance to our vector. We have reached the point where we can now add the ability to enumerate over our vector. All the scaffolding is in place so let’s add that capability.

int capacity() const
{
  return cameras_->capacity();
}

iterator begin() { return cameras_->begin(); }
iterator end() { return cameras_->end(); }

const_iterator begin() const { return cameras_->begin(); }
const_iterator end() const { return cameras_->end(); }
const_iterator cbegin() const { return cameras_->cbegin(); }
const_iterator cend() const { return cameras_->cend(); }

That’s it. We now have the ability to build and iterate over the devices on our system. Let’s see how we would go about using that. To test it, I created a little console application that I used to call my code. It’s as easy as this:

#include "stdafx.h"
#include "cameras.h"
#include 

int main()
{
const auto devices = std::make_shared();
for (auto &dev : *devices)
{
cout <get_information()->dump_diagnostic();
}
return 0;
}

This is what it looks like on my system (running a web camera and a separate RealSense camera).

real-sense-images

Advertisements

Solving the RealSense platform conundrum

When I develop RealSense applications I invariably use Visual Studio and C# to write my applications. If you’ve done any RealSense development in .NET, you’re aware that they have a 32 bit and 64 bit version of their assemblies and you’ve probably had to deal with the conundrum about making your application support 64 bit or 32 bit assembly references. In most of the examples I have seen, the developers plump for adding references to only one platform – which is a real shame. With just a little bit of Visual Studio trickery, you can use target both platforms – in this post, I’m going to introduce you to a little command line tool that I wrote that sets your apps up to support x86 and x64.

Basically, what we’re going to do here is copy the relevant libs files from the Realsense SDK and copy them into a Libs folder in the solution folder. Obviously, we need to reference these files so the code will create x86 and x64 references inside our csproj files (and create the matching entries in the solution as well). As only one of the files is a .NET file, we need to copy the unmanaged DLL it relies on into the output folder as well. This last part is done by creating a post build event to copy the file over on successful completion of the build.

Using it is pretty straightforward – you’ll need to change the root folder for your RealSense installation inside the .config file. Look for the RSSDK key and enter your root directory – in my case, it’s C:\Intel\RSSDK, so when the code is running it uses this to build up the C:\Intel\RSSDK\bin\win32 and C:\Intel\RSSDK\bin\x64 folders. So, you need to make sure you change this value to point to the directory immediately above the bin folder.

When you run the application (it’s a console application so you’re best off running it in a command window), pass in the name of the directory that you want to update to RealSense – the code looks for all solution files and csproj files from the directory you pass in – don’t worry about the nesting level of these files, the code effectively walks the tree looking for these files. So, when I wanted to add RealSense to all of the solution and project files under C:\Dev\TestRealsenseMaker, I ran the command MakeRealsense C:\Dev\TestRealsenseMaker. And that’s it. That’s all I needed to do to make my files RealSense ready. Happy coding.

MakeRealsense.zip

Note: WordPress doesn’t like zip files, so you’ll need to rename the file from MakeRealsense.zip.doc to MakeRealsense.zip when you have downloaded it.

Sensing the future with WPF

This post is a look into a new library that I’m writing that’s intended to make life easier for WPF developers working with Intel RealSense devices. As many of you may know, I’ve been involved with the RealSense platform for a couple of years now (back from when it was called the Perceptual Computing SDK). When I develop samples with it, I tend to use WPF as my default development experience, and the idea of hooking up the Natural User Interface capabilities of RealSense devices with the NUI power of WPF in an easy to use package is just too good to resist. On top of this, I still strongly believe in WPF and it will take a lot to remove me from developing desktop applications with it because it is just so powerful.

To this end, I have started developing a library called RealSenseLight that will enable WPF developers to easily leverage the power of RealSense without having to worry about the implementation details. While it”s primarily aimed at WPF developers, the functionality available will be usable from other C# (Windows Desktop) applications, so hooking into Console applications will certainly be possible.

One of the many decisions I’ve taken is to allow configuration of features to be set via Fluent interface, so it’s possible to do things like this:

RealSenseApplication.Uses(new EmotionDetectionConfiguration())
  .Uses(SpeechRecognition.DefaultConfiguration().ChangePitch(0.8))
  .Start();

ViewModels will be able to hook into RealSense using convenient interfaces that abstract the underlying implementations. There’s no need to call Enable… to enable a RealSense capability. The simple fact of integrating a concrete implementation means that the feature is automatically available. The following example demonstrates what an IoC resolved implementation looks like:

public class EmotionViewModel : ViewModelBase
{
  private IEmotion _emotion;
  public EmotionViewModel(IEmotion emotion)
  {
    emotion.OnUserHappy(user =&gt; System.Debug.WriteLine(&quot;{0} is happy&quot;, user.DetectedUser));
  }
}

The library will provide the ability to do things such as pause/resume individual RealSense capabilities, identify and choose from the relevant RealSense compatible devices, although this does require identifying up front, what the different aspects are you’re interested in because it uses these to evaluate the devices that meet these capabilities.

I’m still fleshing out what the whole interface will look like, so all of the features haven’t been determined yet, but I will keep posting my designs and a link to the repo once I have it in a state where it’s ready for an initial commit.

Getting a RealSense of my status

Long time readers will have realised that I have been spending a lot of time with the technology that was formally known as Perceptual Computing (PerC). You may also know that this technology is now known as RealSense and that it will be rolling out to a device near you soon. What you might not know is that I’m currently writing a course on this technology for Pluralsight. As part of writing this course, I’ve been creating a few little wrapper utilities that will make your life easier when developing apps with the SDK.

In this post, I’m going to show you a handy little method for working with API methods. Pretty much every RealSense API method returns a status code to indicate whether or not it was successful. Now, it can get pretty tedious writing code that looks like this:

pxcmStatus status = Session.CreateImpl<PXCMVoiceRecognition>
  (PXCMVoiceRecognition.CUID, out voiceRecognition);
if (status < pxcmStatus.pxcmStatus.PXCM_STATUS_NO_ERROR)
{
  throw new InvalidStatusException("Could not create session");
}
status = _voiceRecognition.QueryProfile(out pInfo);
if (status < pxcmStatus.pxcmStatus.PXCM_STATUS_NO_ERROR)
{
  throw new InvalidStatusException("Could not query profile");
}

As you can imagine, the more calls you make, the more status checks you have to do. Well, I like to log information about what I’m attempting to do and what I have successfully managed to do, so this simple method really helps to write information about the methods being invoked, and to throw an exception if things go wrong.

public void PipelineInvoke(Func<pxcmStatus> pipelineMethod, string loggingInfo = "")
{
  if (!string.IsNullOrWhiteSpace(loggingInfo))
  {
    Debug.WriteLine("Start " + loggingInfo);
  }
  pxcmStatus status = pipelineMethod();
  if (status < pxcmStatus.PXCM_STATUS_NO_ERROR)
  {
    throw new InvalidStatusException(loggingInfo, status);
  }
  if (!string.IsNullOrWhiteSpace(loggingInfo))
  {
    Debug.WriteLine("Finished " + loggingInfo);
  }
}

This makes it easier to work with the API and gives us code that looks like this:

PipelineInvoke(() => 
  Session.CreateImpl<PXCMVoiceRecognition>(PXCMVoiceRecognition.CUID, 
  out _voiceRecognition), "creating voice recognition module");

And this is what InvalidStatusException looks like:

public class InvalidStatusException : Exception
{
  public InvalidStatusException(string alertMessage, pxcmStatus status)
  {
    Message = alertMessage;
    Status = status;
  }
  public string Message { get; private set; }
  public pxcmStatus Status { get; private set; }
}

Over the course of the next couple of months, I’ll share more posts with you showing the little tricks and techniques that I use to make working with the RealSense SDK a real joy.