100 Days of TypeScript (Day 1)

Introducing TypeScript

It’s been a while since I’ve had the chance to write in this blog and I really wanted to come up with something different for me. If you’re a long time follower of mine, you probably know that I wrote a book on TypeScript a couple of years ago, so I thought it would be fun for me to embark on one of those 100 Days of types of blog and write a series introducing you to TypeScript if you’ve not used it before; give you a brush up on things you already know if you’ve used it, and maybe introduce you to some new things that you might not have come across before.

Note: This blog series isn’t going to be a continuous 100 days of posting, it’s 100 days by the end.

So, what is TypeScript? If you go to the TypeScript website, the tagline is “TypeScript is JavaScript with syntax for types”. What does that actually mean?

TypeScript was originally built out of the idea that writing high quality JavaScript was difficult. While that’s a subjective opinion, TypeScript rapidly evolved, giving us the ability to use up and coming JavaScript features via polyfills (don’t worry about this, we’ll come back to this in a future post). TypeScript now ranks as one of the most popular languages around.

Getting Started

Okay, you can develop TypeScript apps online in the playground, but I really recommend installing it locally. To do that, you’re going to need to use something like npm, yarn or pnmpm. If you don’t have npm installed, you need to install Node to get it. Assuming you have installed Node, you can install TypeScript with the following command (depends on the package manager you’re using).

npm install -g typescript

It’s time to write our first TypeScript application (all code is available in github).

This is going to be a simple program to add two numbers together. I’m going to use the TypeScript compiler to set some things up ready for when I want to compile my TypeScript code.

tsc --init

This has created a file called tsconfig.json which sets up compiler options that determine what our JavaScript will look like when we’ve compiled it from TypeScript; oh wait, didn’t I mention that TypeScript compiles down to JavaScript? A quick note – I pretty much always use Visual Studio Code for editing my TypeScript; it’s a great choice if you’ve not used it before.

I’m going to add a file called day1.ts to add my TypeScript to add my first piece example. The .ts extension tells us that this is a TypeScript file. If you have used JavaScript before, the code will look familiar to you. This is what our add function would look like as a JavaScript method.

function add(number1, number2) {
    return number1 + number2;
}

I said that I wanted the add function to be able to add two numbers. I don’t want it to add two strings together or a date and a string. This is where the first strength of TypeScript really comes into its own, and where the TypeScript tagline makes sense. I am going to constrain the function to accept numbers and return a number using the following syntax.

function add(number1: number, number2: number): number {
    return number1 + number2;
}

If I attempt to pass something that’s not a number to either of the parameters, I won’t be able to compile the code. I can’t pass “special” values such as undefined or null to either of the parameters. In other words, I have just written something that will protect me from myself.

In order to test my code, I’m going to call the function and pass the output to the console window like this.

console.log(add(10, 20));

After saving the file, I want to “compile” the TypeScript code so that it turns into JavaScript. To do this, I simply run the tsc command with no parameters. This picks up the contents of the tsconfig.json file and uses that to control how the file is compiled. (The tsc command here should be run in the same directory as the tsconfig.json file).

Now that I have compiled my code, I can run it using the following command.

node day1.js

And that’s it; that’s the first TypeScript program. In day 2, we are going to look at how to change our function over so that it sits inside a class; which will let us see how we can start to leverage object oriented concepts to build reusable blocks.

FluentRealSense – the first steps to a simpler RealSense

Those who know me are aware that I have a long term association (nay, let’s say it for what it is, love affair) with the RealSense platforms from Intel. If you’ve been following developments in that space for any length of time, you’re probably aware that things have moved on a lot and the cameras are available in all sorts of devices and are no longer just limited to the Windows platform. This shift of emphasis means that Intel have moved away from concentrating on the old monolithic RealSense SDK and have moved to supporting a new open source API (available here: https://github.com/IntelRealSense/librealsense).

I have spent a period of time working with this API as it has evolved and have started “wrapping” it to make it a little bit friendlier to use (in my opinion) for people who don’t want to get bogged down with the “nitty gritty” of how to get devices, and so on. This work has been in C++, the language the API is available in, and I am going to cover, over a series of posts, how this library evolved. I’ll show you the code I’ve written at each step so that you can also see how I have gone about relearning modern C++ (bearing in mind it has been 18 years since I last worked with it in anger) so you will definitely find that early iterations of the code are naive, at best. My aim with this library was to make it as fluent as possible, hence it’s name, FluentRealSense.

An upfront note. Early iterations of this codebase have been done entirely on Windows running Visual Studio 2017. I chose this particular environment because, a) Visual Studio is, to my mind, the finest IDE around and b) because 2017 supports porting your C++ code directly over to Linux. This lets me leverage the rapid turnaround time I’m used to in my dev environment and the ease of deploying to my intended targets later on. Thank you Microsoft for keeping this old developer happy.

Let’s start with the first iteration. The first thing I wanted the code to do was to provide me with the ability to iterate over all the cameras on my system and print out some diagnostic information. This meant that my code would be broken down initially into three classes:
information: This class reads information from the API about a single camera and returns it as a string.
camera: This is a single device. Initially, it’s just going to instantiate and provide access to the information class.
cameras: The entry point for consuming applications, this class is responsible for finding out about and managing access to each camera; this class is also enumerable so that the calling code can quickly iterate over each camera.

Let’s start off by looking at the information class. First we’re going to add some includes and namespaces:

#pragma once
#include "hpp/rs_context.hpp"

using namespace std;
using namespace rs2;

class information
{
public:
class information() {}
~information() {}
}

That’s all very straightforward and nothing you won’t have seen before. Let’s start fleshing this out.

The first thing we’re going to need is a RealSense device to get information from – this is going to be passed in so let’s add a member to store it and replace our constructor with this:

explicit information(const device device) : device_(device) {}
:::
private:
device device_;

At this stage, our code looks like this:

#pragma once
#include "hpp/rs_context.hpp"

using namespace std;
using namespace rs2;

class information
{
public:
explicit information(const device device) : device_(device) {}
~information() {}

private:
device device_;
}

I have to say, at this point, I love the improvements to instantiating members that C++ now provides. This is a wonderful little innovation.

Now, in order to get the information out of the API, I’m going to add a handy little private helper method. This makes use of the fact that the API exposes this information via an rs2_camera_info enumeration:

const char* get_info(const rs2_camera_info info) const
{
if (!device_.supports(info))
{
return "Not supported";
}
return device_.get_info(info);
}

This is the first point that our code is going to add any value. Whenever we call get_info against the underlying device, we have to check that that particular type of info can be retrieved for that device. It’s very easy to write code that makes assumptions that each camera supports exactly the same set of properties here so the underlying call could throw an exception. To get around this, our code checks to see if the device supports that particular rs2_camera_info type and returns “Not supported” if it doesn’t. It is so easy to forget that you should always pair the get_info with the supports, so this helper method removes the need to remember that.

So, how do we use this? Well, with some public methods like these:

const char* name() const
{
return get_info(RS2_CAMERA_INFO_NAME);
}

const char* serial_number() const
{
return get_info(RS2_CAMERA_INFO_SERIAL_NUMBER);
}

const char* port() const
{
return get_info(RS2_CAMERA_INFO_PHYSICAL_PORT);
}

const char* firmware_version() const
{
return get_info(RS2_CAMERA_INFO_FIRMWARE_VERSION);
}

const char* debug_opCode() const
{
return get_info(RS2_CAMERA_INFO_DEBUG_OP_CODE);
}

const char* advanced_mode() const
{
return get_info(RS2_CAMERA_INFO_ADVANCED_MODE);
}

const char* product_id() const
{
return get_info(RS2_CAMERA_INFO_PRODUCT_ID);
}

const char* camera_locked() const
{
return get_info(RS2_CAMERA_INFO_CAMERA_LOCKED);
}

string dump_diagnostic() const
{
string text = "\nDevice Name: ";
text += name();
text += "\n Serial number: ";
text += serial_number();
text += "\n Port: ";
text += port();
text += "\n Firmware version: ";
text += firmware_version();
text += "\n Debug op code: ";
text += debug_opCode();
text += "\n Advanced Mode: ";
text += advanced_mode();
text += "\n Product id: ";
text += product_id();
text += "\n Camera locked: ";
text += camera_locked();

return text;
}

We now have our complete information class. It looks like this:

#pragma once
#include "hpp/rs_context.hpp"

using namespace std;
using namespace rs2;

class information
{
public:
explicit information(const device device) : device_(device) {}
~information()
{
}

const char* name() const
{
  return get_info(RS2_CAMERA_INFO_NAME);
}

const char* serial_number() const
{
  return get_info(RS2_CAMERA_INFO_SERIAL_NUMBER);
}

const char* port() const
{
  return get_info(RS2_CAMERA_INFO_PHYSICAL_PORT);
}

const char* firmware_version() const
{
  return get_info(RS2_CAMERA_INFO_FIRMWARE_VERSION);
}

const char* debug_opCode() const
{
  return get_info(RS2_CAMERA_INFO_DEBUG_OP_CODE);
}

const char* advanced_mode() const
{
  return get_info(RS2_CAMERA_INFO_ADVANCED_MODE);
}

const char* product_id() const
{
  return get_info(RS2_CAMERA_INFO_PRODUCT_ID);
}

const char* camera_locked() const
{
  return get_info(RS2_CAMERA_INFO_CAMERA_LOCKED);
}

string dump_diagnostic() const
{
  string text = "\nDevice Name: ";
  text += name();
  text += "\n Serial number: ";
  text += serial_number();
  text += "\n Port: ";
  text += port();
  text += "\n Firmware version: ";
  text += firmware_version();
  text += "\n Debug op code: ";
  text += debug_opCode();
  text += "\n Advanced Mode: ";
  text += advanced_mode();
  text += "\n Product id: ";
  text += product_id();
  text += "\n Camera locked: ";
  text += camera_locked();
  return text;
}

private:
  const char* get_info(const rs2_camera_info info) const
  {
    if (!device_.supports(info))
    {
      return "Not supported";
    }
    return device_.get_info(info);
  }
  device device_;
};

The next thing we have to do is write a class that represents a single RealSense camera. There’s not much to this class, at the moment, so let’s look at it in its entirety.

#pragma once
#include "hpp/rs_context.hpp"
#include "information.h"

using namespace std;
using namespace rs2;

class camera
{
public:
  explicit camera(const device dev) : information_(make_shared(dev)) {}

  ~camera()
  {
  }

  shared_ptr get_information() const
  {
    return information_;
  }

private:
  shared_ptr information_;
};

I did say this class was pretty light at the moment. The class accepts a single RealSense device which is used to instantiate the information class. We provide one method which is used to get the instance of the information class. That’s it so far.

Finally, we come to the entry point of our code, the cameras class. This class let’s us enumerate all of the cameras on our system and access the functions inside. As usual, we’ll start off with the definition:

#pragma once
#include 
#include 
#include "camera.h"
#include "hpp/rs_context.hpp"

using namespace std;
using namespace rs2;

class cameras
{
public:
  cameras() {}
  ~cameras() {}
}

As you will remember, I said that I wanted the cameras to be enumerable so I need to do some upfront declarations:

using cameras_t = vector<shared_ptr>;
using iterator = cameras_t::iterator;

using const_iterator = cameras_t::const_iterator;
[/source]

With these in place, I can now start to add the ability to enumerate over the camera instances. Before I do that, though, it’s time to introduce something new. In the preceding code, we saw that the RealSense camera was represented as a device that we passed into the relevant constructors. The question is, how did we get that device in the first place? Well, that’s down to the API providing a context that allows us to access these devices. So, let’s add a member to store a vector of camera instances and then build in the method to get the list of devices.

cameras() : cameras_(make_shared())
{
  context context;
  // Iterate over the devices;
  auto devices = context.query_devices();
  for (const auto dev : devices)
  {
    const auto cam = make_shared(dev);
    cameras_->push_back(cam);
  }
}

private:
  shared_ptr cameras_;

There’s nothing complicated in that code. We get the devices from the context using query_devices and iterate over each one, adding a new camera instance to our vector. We have reached the point where we can now add the ability to enumerate over our vector. All the scaffolding is in place so let’s add that capability.

int capacity() const
{
  return cameras_->capacity();
}

iterator begin() { return cameras_->begin(); }
iterator end() { return cameras_->end(); }

const_iterator begin() const { return cameras_->begin(); }
const_iterator end() const { return cameras_->end(); }
const_iterator cbegin() const { return cameras_->cbegin(); }
const_iterator cend() const { return cameras_->cend(); }

That’s it. We now have the ability to build and iterate over the devices on our system. Let’s see how we would go about using that. To test it, I created a little console application that I used to call my code. It’s as easy as this:

#include "stdafx.h"
#include "cameras.h"
#include 

int main()
{
const auto devices = std::make_shared();
for (auto &dev : *devices)
{
cout <get_information()->dump_diagnostic();
}
return 0;
}

This is what it looks like on my system (running a web camera and a separate RealSense camera).

Solving the RealSense platform conundrum

When I develop RealSense applications I invariably use Visual Studio and C# to write my applications. If you’ve done any RealSense development in .NET, you’re aware that they have a 32 bit and 64 bit version of their assemblies and you’ve probably had to deal with the conundrum about making your application support 64 bit or 32 bit assembly references. In most of the examples I have seen, the developers plump for adding references to only one platform – which is a real shame. With just a little bit of Visual Studio trickery, you can use target both platforms – in this post, I’m going to introduce you to a little command line tool that I wrote that sets your apps up to support x86 and x64.

Basically, what we’re going to do here is copy the relevant libs files from the Realsense SDK and copy them into a Libs folder in the solution folder. Obviously, we need to reference these files so the code will create x86 and x64 references inside our csproj files (and create the matching entries in the solution as well). As only one of the files is a .NET file, we need to copy the unmanaged DLL it relies on into the output folder as well. This last part is done by creating a post build event to copy the file over on successful completion of the build.

Using it is pretty straightforward – you’ll need to change the root folder for your RealSense installation inside the .config file. Look for the RSSDK key and enter your root directory – in my case, it’s C:\Intel\RSSDK, so when the code is running it uses this to build up the C:\Intel\RSSDK\bin\win32 and C:\Intel\RSSDK\bin\x64 folders. So, you need to make sure you change this value to point to the directory immediately above the bin folder.

When you run the application (it’s a console application so you’re best off running it in a command window), pass in the name of the directory that you want to update to RealSense – the code looks for all solution files and csproj files from the directory you pass in – don’t worry about the nesting level of these files, the code effectively walks the tree looking for these files. So, when I wanted to add RealSense to all of the solution and project files under C:\Dev\TestRealsenseMaker, I ran the command MakeRealsense C:\Dev\TestRealsenseMaker. And that’s it. That’s all I needed to do to make my files RealSense ready. Happy coding.

MakeRealsense.zip

Note: WordPress doesn’t like zip files, so you’ll need to rename the file from MakeRealsense.zip.doc to MakeRealsense.zip when you have downloaded it.

Sensing the future with WPF

This post is a look into a new library that I’m writing that’s intended to make life easier for WPF developers working with Intel RealSense devices. As many of you may know, I’ve been involved with the RealSense platform for a couple of years now (back from when it was called the Perceptual Computing SDK). When I develop samples with it, I tend to use WPF as my default development experience, and the idea of hooking up the Natural User Interface capabilities of RealSense devices with the NUI power of WPF in an easy to use package is just too good to resist. On top of this, I still strongly believe in WPF and it will take a lot to remove me from developing desktop applications with it because it is just so powerful.

To this end, I have started developing a library called RealSenseLight that will enable WPF developers to easily leverage the power of RealSense without having to worry about the implementation details. While it”s primarily aimed at WPF developers, the functionality available will be usable from other C# (Windows Desktop) applications, so hooking into Console applications will certainly be possible.

One of the many decisions I’ve taken is to allow configuration of features to be set via Fluent interface, so it’s possible to do things like this:

RealSenseApplication.Uses(new EmotionDetectionConfiguration())
  .Uses(SpeechRecognition.DefaultConfiguration().ChangePitch(0.8))
  .Start();

ViewModels will be able to hook into RealSense using convenient interfaces that abstract the underlying implementations. There’s no need to call Enable… to enable a RealSense capability. The simple fact of integrating a concrete implementation means that the feature is automatically available. The following example demonstrates what an IoC resolved implementation looks like:

public class EmotionViewModel : ViewModelBase
{
  private IEmotion _emotion;
  public EmotionViewModel(IEmotion emotion)
  {
    emotion.OnUserHappy(user => System.Debug.WriteLine("{0} is happy", user.DetectedUser));
  }
}

The library will provide the ability to do things such as pause/resume individual RealSense capabilities, identify and choose from the relevant RealSense compatible devices, although this does require identifying up front, what the different aspects are you’re interested in because it uses these to evaluate the devices that meet these capabilities.

I’m still fleshing out what the whole interface will look like, so all of the features haven’t been determined yet, but I will keep posting my designs and a link to the repo once I have it in a state where it’s ready for an initial commit.

Getting a RealSense of my status

Long time readers will have realised that I have been spending a lot of time with the technology that was formally known as Perceptual Computing (PerC). You may also know that this technology is now known as RealSense and that it will be rolling out to a device near you soon. What you might not know is that I’m currently writing a course on this technology for Pluralsight. As part of writing this course, I’ve been creating a few little wrapper utilities that will make your life easier when developing apps with the SDK.

In this post, I’m going to show you a handy little method for working with API methods. Pretty much every RealSense API method returns a status code to indicate whether or not it was successful. Now, it can get pretty tedious writing code that looks like this:

pxcmStatus status = Session.CreateImpl<PXCMVoiceRecognition>
  (PXCMVoiceRecognition.CUID, out voiceRecognition);
if (status < pxcmStatus.pxcmStatus.PXCM_STATUS_NO_ERROR)
{
  throw new InvalidStatusException("Could not create session");
}
status = _voiceRecognition.QueryProfile(out pInfo);
if (status < pxcmStatus.pxcmStatus.PXCM_STATUS_NO_ERROR)
{
  throw new InvalidStatusException("Could not query profile");
}

As you can imagine, the more calls you make, the more status checks you have to do. Well, I like to log information about what I’m attempting to do and what I have successfully managed to do, so this simple method really helps to write information about the methods being invoked, and to throw an exception if things go wrong.

public void PipelineInvoke(Func<pxcmStatus> pipelineMethod, string loggingInfo = "")
{
  if (!string.IsNullOrWhiteSpace(loggingInfo))
  {
    Debug.WriteLine("Start " + loggingInfo);
  }
  pxcmStatus status = pipelineMethod();
  if (status < pxcmStatus.PXCM_STATUS_NO_ERROR)
  {
    throw new InvalidStatusException(loggingInfo, status);
  }
  if (!string.IsNullOrWhiteSpace(loggingInfo))
  {
    Debug.WriteLine("Finished " + loggingInfo);
  }
}

This makes it easier to work with the API and gives us code that looks like this:

PipelineInvoke(() => 
  Session.CreateImpl<PXCMVoiceRecognition>(PXCMVoiceRecognition.CUID, 
  out _voiceRecognition), "creating voice recognition module");

And this is what InvalidStatusException looks like:

public class InvalidStatusException : Exception
{
  public InvalidStatusException(string alertMessage, pxcmStatus status)
  {
    Message = alertMessage;
    Status = status;
  }
  public string Message { get; private set; }
  public pxcmStatus Status { get; private set; }
}

Over the course of the next couple of months, I’ll share more posts with you showing the little tricks and techniques that I use to make working with the RealSense SDK a real joy.

Haswell – Intel SDP Unit (Software Developer Preview) – a death in the family

Okay, that’s possibly a bit too melodramatic a title, but that’s almost what it felt like when my Ultrabook decided to pack in and shuffle off to the great gig in the sky. This post will contain no screenshots for reasons that will soon become abundantly clear.

Some context first, I’ve been using the Ultrabook pretty much continuously since I got it. It was my default, go-to, day to day development box. Yup, that’s a lot of different ways of saying that I was using the Ultrabook very heavily. So heavily, in fact, that I was using it as the workhorse for developing my Synxthasia project for Intel ; a Theramin(ish) type of music application which translated gestures in 3D space into sound and visuals – it even took your picture at intervals and imposed that into the visuals; I was particularly proud of the fact that it gave you the ability to alter the “shape” of the sound to simulate a wah effect just by using your mouth. The SDP really is an excellent device for development tasks like this.

So, about a week before I was due to submit this to Intel; with the app heavily in the polishing stages, the Ultrabook died on me – taking 8 days of code that hadn’t been committed back to source control (I have no excuses really as I should have been committing this to source control as I went along). Remember that I said that it had the habit of giving the low battery warning and then switching off, well this is exactly what happened. Anyhoo, I plugged it in and started it back up – all was fine and dandy here, but I noticed that Windows was reporting that there was no battery present. Well, I couldn’t leave the Ultrabook permanently plugged in and I needed to go to a client site – I unplugged the unit and set off. When I tried to power it back on later on, it refused to start – all that happened was the power button LED flashed blue and that was it.

I got in touch with contacts in Intel and they provided some first line support options which included opening the unit and disconnecting the batteries. Unfortunately, this hasn’t worked – the unit still has all the vitality of the Norwegian Blue. Intel has offered to replace the device but as I have been away from the physical unit for a while, I’ve been unable to take them up on their kind offer. Once I get back home, I will avail myself of their help because the device itself really is that good.

What has become clear to me over the last year or so, is just how much Intel cares about the feedback it gets. I cannot stress enough how they have listened to reviewers and developers like myself, and how they have sought to incorporate that feedback into their products. As a developer, it’s a genuine pleasure for me to deal with them as they really do seem to understand what my pain points are, and they provide a heck of a lot of features to help me get past those problems. Why do I say this? Well, I was lucky enough to have an SDP from last year to use and while it was a good little device, there were one or two niggles that really got to me as I used it – the biggest problem, of course, being the keyboard. The first SDP just didn’t have a great keyboard. The second issue was that the first SDP also felt flimsy – I always felt that opening the screen was a little fragile and that I could do damage if I opened it too vigorously. Well, not this time round – the updated version of the SDP has a fantastic keyboard, and feels as robust as any laptop/Ultrabook that I’ve used, and all this in a slimmer device. You see, that’s what I mean by Intel listening. I spend a lot of time at the keyboard and I need to feel that it’s responsive – and this unit certainly is. The thing is, Intel aren’t selling these units – they are giving them away for developers to try out for free. It would be perfectly understandable if they cut corners to save costs, but it’s patently apparent that they haven’t – they really have tried to give us a commercial level device. Yes, I’ve been unlucky that the Haswell died on me, but given the care from Intel this hasn’t tarnished my opinion of it.

So, does the death of the device put me off the Haswell? Of course it doesn’t. For the period I have been using it, it has been a superb workhorse. It combines good looks with performance with great battery life. It’s been absolutely outstanding and I wouldn’t hesitate to recommend it. Needless to say, I’m delighted with the help and support that I’ve had from Intel – it certainly hasn’t been any fault of theirs that I’ve been unable to return the device for them to replace. I’d also like to thank Rick Puckett at Intel for his help and patience, as well as Carrie Davis-Sydor and Iman Saad at DeveloperMedia for helping hook me up with the unit in the first place and their help when the Haswell shuffled off the mortal coil.

Haswell – Intel SDP Unit (Software Developer Preview) – The Keyboard fights back

Well, since my initial review of the Ultrabook, I’ve been using it for pretty much all my computing needs. From writing C++ and C# code, through standard surfing and the likes, through to heavier duty 3D work, and the Haswell just keeps on going.

First of all, let me just say that the keyboard on this generation of Intel SDP is a whole lot better than the one that shipped with last years SDP. This one actually feels solid and responsive, and it features a nice little backlight. Intel really has upped the game for what are effectively demo units here. My only, admittedly minor, niggle is the fact that the keyboard is designed for the US market. When I change it to the UK settings, the \ key is missing.

Writing code using this Ultrabook is a pleasant experience. The screen resolution is good, and the screen itself is bright and very pleasant to use. Unfortunately, the 4GB memory means that there’s sometimes a slow down when compiling a C++ application from scratch using Visual Studio 2012. Once you’re past the first compilation though, the compile process is responsive enough, so this isn’t a huge issue. As I’m a heavy user of Visual Studio and WPF, I’m pleased to see that the responsiveness of the WPF designer window is nice and snappy – there’s no hold up in the system making me wish I was using a different machine. Even the dodgy upper case menus in Visual Studio don’t distract me while I’m looking at this glorious screen.

One thing I’ve tried to do is keep the settings of the machine as default as possible (except for the keyboard layout – that’s’ a compromise too far). This has allowed me to test the claims that the Haswell gives me long battery life. Well, it’s true – using it for a mixture of DirectX development, 3D work and WPF development has allowed me to get just over 7 hours on a single charge. It’s not quite the potential 12 hours, but my day to day usage is pretty atypical, so I wouldn’t be unduly worried about the battery life if I were looking to use this on a really long journey. The reason that I mentioned the defaults is because Windows 8 suddenly tells me that I’ve got about 5% battery life left, and then it shuts down – there’s not enough time for me to get the unit plugged in then.

Now Pete – surely it can’t be all sweetness and light, can it? Where are the niggles that must exist here, or are you playing nicey-nice with Intel here? Well, there is one thing that really gets me annoyed and that’s the Trackpad. If I click on the left hand side of the Trackpad (anywhere on the left), I can trigger a click – and that’s exactly what I’d expect. Clicking or tapping on the right hand side doesn’t trigger anything. This has caused me quite a bit of frustration. It’s my issue to deal with though, as the Trackpad is a multi touch unit so it’s something I’m going to have to work on.

You may wonder what applications I typically have running on the Ultrabook, to see how this compares to what you would use. Well, on a day to day basis, I typically have up to 4 instances of Visual Studio 2012 running simultaneously. I also have a couple of instances of Expression Blend running, as well as Chrome, with a minimum of 10 to 12 fairly heavy web pages in them (if you’re really interested, these pages are Facebook, Code Project – several instances covering different forums and articles,, GMail, Twitter, The Daily Telegraph and nufcblog.org). I also usually have Word, Excel, Photoshop, Huda and Cinema 4D running, along with Intel’s Perceptual Computing SDK and Creative Gesture camera. As you can see, that’s a lot of heavy duty processing going on there, and this machine copes with them admirably.

Okay, to whet your appetite – I’ve been using the Ultrabook, off the charger, just editing documents and web pages for the last two hours, and I still have 91% battery life left. That’s absolutely incredible.

 

Haswell – Intel SDP Unit (Software Developer Preview) – 1st impressions

The reveal

Bonk, bonk bonk bonk bong….

That’s the sound that greeted me when I opened the box from Intel™ featuring a developer preview Haswell Ultrabook™, and there’s no geek in the world who wouldn’t want to repeat that sound, so after opening and closing the case for ten minutes or so, I decided to actually get the Ultrabook™ out of the box.

Okay, this is the Ultrabook™ packaging (outside of the cardboard it was shipped in). Little did I know the simple joy I was to receive upon opening this box.

I’ve opened up the packaging, and this is the sight that greets me. I’m keen to start the unit, but I want to see what other goodies the box has inside.

Is that a carry case for the Ultrabook™? Nope, it’s just packing material to keep the unit from getting scratched.

Ahh, that’s more like it. Cables and a USB drive (containing the Windows reinstallation media). The reflection in this image is from the plastic covering that protects this layer.

Okay, the plastic is off and I can see the quick start guide as well as the power supply and USB drive. That PSU is tiny. Consider me seriously impressed.

What have we here? Ah, Mini HDMI to Full, to VGA and a USB to Ethernet. Thoughtful of them to supply this.

Well, there’s the keyboard. Nice.

After starting the machine up, it’s a simple matter to configure Windows™ 8. Windows really suits a machine like this.

Day to day

Well, the first thing to do with a machine like this is to take it out for a drive. In my case, this means installing Visual Studio Ultimate 2012. As you might expect, I have several computers, so I have a lot of reference there on how long this actually takes to install. On my old I7 with a spindle hard drive, it took up to an hour to install. The question then, is how long would it take to install on this machine with the same feature set. Start to finish, the whole process took 8 minutes. Yup, you read that right, 8 minutes start to finish. I couldn’t ask for a better initial impression than that.

The good

This unit is fast. Oh, it’s so very fast. Okay, the boot up time is a tad slower than my I7 Lenovo™ Yoga 13, (by less than half a second), but once you actually get going, the machine is very responsive. While it’s a fairly entry level I5, it is more than capable of coping with my day to day development tasks. Opening up a XAML page in Visual Studio 2012 and showing it with both the designer and code view open, the code view updated Intellisense as I typed, and the designer refreshed itself snappily – those of you who develop XAML based apps know how slow this can be – it was an absolute joy to behold.

The unit is running Windows 8 Pro (Build 9200), and Windows works well on it. The screen is responsive to the touch, and Windows feels snappy and responsive.

The machine is light. Maybe it’s not Mac Air light, but it’s heading in the right direction, and it’s thin. If you were going to use this only on your desktop, this wouldn’t matter but if, like me, you travel a lot, you’ll really appreciate them.

That battery life. Oh boy, 6 hours spent doing a lot of fairly intensive development and the battery still has plenty of life to give yet. This really has hit the sweetspot with regards to providing me with portability. I can easily see myself whiling away journeys using this.

The niggles

This is going to sound churlish considering I’ve received this unit as a “favour”, but I do wish it had more than 4GB of RAM and a bigger hard drive than it has (a 128GB SSD).

The shift keys are on the large side, but I have no doubt I will soon get used to them.

There’s a lot of real estate lost around the screen, but I’ve no doubt that commercial units will take care of this, using all the available space.

It’s hard to put my finger on what the exact problem is, but the keys don’t feel quite right to me. There’s a bit of play to them that doesn’t make them feel that stable. Obviously, as this is not a commercial Ultrabook™, I’m not unduly worried – I would expect a commercial Ultrabook™ to have a more solid feel to the keys.

So, where are we at?

I’ve been asked to write the reviews as unbiased as I possibly can be. This review is my initial impression only. My next review will cover using this machine on a daily basis. As I do a lot of Ultrabook™ related work as part of my daily development routine, this will be my primary device, so I’ll let you know how this compares.

Disclosure of Material Connection:

I received one or more of the products or services mentioned above for free in hope that I would mention it on my blog. Regardless, I only recommend products or services I use personally and believe my readers will enjoy. I am disclosing this in accordance with the Federal Trade Commission’s 16 CFR, Part 255: “Guides Concerning the Use of Endorsements and Testimonials in Advertising.”

Ultimate Coder – The pursuit of Bacon

This is a copy of the post I made on the Intel site here. For the duration of the contest, I am posting a weekly blog digest of my progress with using the Perceptual Computing items. This weeks post shows how Huda has evolved from the application that was created at the end of the fourth week. My fellow competitors are planning to build some amazing applications, so I’d suggest that you head on over to the main Ultimate Coder site and follow along with their progress as well. I’d like to take this opportunity to wish them good luck.

Bring me the Bacon.

Rather than doing a day by day update this week, I’ve waited until near the end of the week to cover the progress I’ve been making this week. It couldn’t be more timely, given Steve’s update this week as I’d reached a lot of the conclusions he did regarding gesture performance and also layout of the interface (a point he raised elsewhere). Hopefully, having a bit of video footage should showcase where we are with Huda right now.

The first part of this week revolved around finalising the core saving and loading of photographs. As I revealed last week, the filters aren’t applied directly to the image; rather, they are associated with the image in a separate file. The code I showed last week demonstrated that each filter would be responsible for serialising itself, so the actual serialisation graph code should be relatively trivial.

using AForge.Imaging.Filters;
using GalaSoft.MvvmLight.Messaging;
using Huda.Messages;
using Huda.Transformations;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;
using System.Threading.Tasks;
namespace Huda.Model
{
public sealed class PhotoManager
{
private Dictionary<string, List<IImageTransform>> savedImageList = new Dictionary<string, List<IImageTransform>>();
private readonly object SyncLock = new object();
private string savedirectory;
private string path;
public PhotoManager()
{
}
public List<IImageTransform> GetFilters(string originalImage)
{
if (!savedImageList.ContainsKey(originalImage))
{
lock (SyncLock)
{
if (!savedImageList.ContainsKey(originalImage))
{
savedImageList.Add(originalImage, new List<IImageTransform>());
Save();
}
}
}
return savedImageList[originalImage];
}
private void AddFilter(string key, IImageTransform filter)
{
var filters = GetFilters(key);
lock (SyncLock)
{
filters.Add(filter);
Save();
}
}
private void Save()
{
if (savedImageList == null || savedImageList.Count == 0) return;
using (FileStream fs = new FileStream(path, FileMode.Create, FileAccess.Write))
{
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(fs, savedImageList);
}
}
public void Load()
{
savedirectory = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData), @"Huda");
if (!Directory.Exists(savedirectory))
{
Directory.CreateDirectory(savedirectory);
}
path = Path.Combine(savedirectory, @"Huda.hda");
Messenger.Default.Register<FilterUpdatedMessage>(this, FilterUpdated);
if (!File.Exists(path)) return;
using (FileStream fs = new FileStream(path, FileMode.Open, FileAccess.Read))
{
BinaryFormatter formatter = new BinaryFormatter();
savedImageList = (Dictionary<string, List<IImageTransform>>)formatter.Deserialize(fs);
}
}
private void FilterUpdated(FilterUpdatedMessage filter)
{
if (filter == null) throw new ArgumentNullException("filter");
AddFilter(filter.Key, filter.Filter);
}
}
}

While I could have created a custom format to wrap the Dictionary there, it would have been overkill for what I was trying to achieve. The reason for me using a Dictionary in this class is that it’s a great way to see if an item is already present – helping to satisfy our requirement to load the filters that we previously applied to the photo.

This class subscribes to all the messages that Huda will raise with relation to filters, which means that we can easily save the state of the filters as and when they are updated. A key tenet of MVVM is that we shouldn’t be introducing knowledge of one ViewModel in another ViewModel, so we use messaging as an abstraction to pass information around. The big advantage of this is that we are able to hook into the same messages here in the model, so we don’t have to worry about which ViewModel is going to update the model. In MVVM, it’s okay to have the link between the ViewModel and the Model, but as we already have this messaging mechanism in place, it’s straightforward for us to just use it here.

Saving and loading the actual photo and filter information is accomplished using a simple BinaryFormatter. As the filters are explicitly serializing themselves, the performance should be acceptable as the formatter doesn’t have to rely on reflection to work out what it needs to save. That’s Pete’s big serializer hint for you this week – it may be easy to let the formatter work out what to serialize, but when your file gets bigger, the load takes longer.

You may wonder why I’m only guarding the Save routine through a lock (note, the lock is actually in the AddFilter method). The Load routine is only called once – at application startup, and it has to finish before the application can continue, so we have this in a thread in the Startup routine – and the splash screen doesn’t disappear until all the information is loaded.

Okay, that’s the core of the photo management in place. Now it’s just a matter of maintaining the filter lists, and that’s accomplished through applying the filters.

You may recall from last weeks code that some of the filters had to convert from BitmapSource to Bitmap (needed for AForge support), but what wasn’t apparent was the fact that we need to convert to a standard image format. Knowing this, I decided to simplify my life by introducing the following helper code:

public static Bitmap To24BppBitmap(this BitmapSource source)
{
return ToBitmap(source, System.Drawing.Imaging.PixelFormat.Format24bppRgb);
}
public static Bitmap ToBitmap(BitmapSource source, System.Drawing.Imaging.PixelFormat format)
{
using (Bitmap bmp = ToBitmap(source))
{
return AForge.Imaging.Image.Clone(bmp, format);
}
}

This enabled me to simplify my filter calls quite considerably. So, the Sepia transformation code, for instance, became this:

public System.Windows.Media.Imaging.BitmapSource Transform(System.Windows.Media.Imaging.BitmapSource source)
{
using (Bitmap bitmap = source.To24BppBitmap())
{
using (Bitmap bmp = new Sepia().Apply(bitmap))
{
return ImageSourceHelpers.FromBitmap(bmp);
}
}
}
[/csharp]

Another thing that should be apparent in that code is that I'm wrapping Bitmap allocations inside using statements. This is because Huda was leaking memory all over the place. Basically, Bitmap objects were being created and not released - this needed taking care of. Fortunately, that's a very simple fix - all it requires was for me to wrap the Bitmap allocations in using statements.

I realised after I posted last weeks article that you might actually like to see the code I'm using to get the gesture to work with the tree. It's incredibly simple and very similar to the POC sample I showed last week. So, if you want to hook the gesture code in, all you need to do is:


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Interactivity;
using System.Windows.Input;
using System.Windows.Controls;
using System.Diagnostics;
using System.Timers;
using System.Windows.Media;
using System.Windows.Threading;
using Goldlight.Perceptual.Sdk;
namespace Huda.Behaviors
{
public class TreeviewGestureItemBehavior : Behavior<TreeView>
{
private bool inBounds;
private Timer selectedTimer = new Timer();
private TreeViewItem selectedTreeview;
private TreeViewItem previousTreeView;
protected override void OnAttached()
{
PipelineManager.Instance.Gesture.HandMoved += Gesture_HandMoved;
selectedTimer.Interval = 2000;
selectedTimer.AutoReset = true;
selectedTimer.Elapsed += selectedTimer_Elapsed;
base.OnAttached();
}
protected override void OnDetaching()
{
PipelineManager.Instance.Gesture.HandMoved -= Gesture_HandMoved;
selectedTimer.Stop();
base.OnDetaching();
}
void selectedTimer_Elapsed(object sender, ElapsedEventArgs e)
{
selectedTimer.Stop();
Dispatcher.Invoke(DispatcherPriority.Normal,
(Action)delegate()
{
if (previousTreeView != null)
{
previousTreeView.IsSelected = false;
}
previousTreeView = selectedTreeview;
selectedTreeview.IsSelected = true;
});
}
void Gesture_HandMoved(object sender, Goldlight.Perceptual.Sdk.Events.GesturePositionEventArgs e)
{
Dispatcher.Invoke((Action)delegate()
{
Point pt = new Point(e.X - ((MainWindow)App.Current.MainWindow).GetWidth() / 2, e.Y - ((MainWindow)App.Current.MainWindow).GetHeight() / 2);
Rect rect = new Rect();
rect = AssociatedObject.RenderTransform.TransformBounds(
new Rect(0, 0, AssociatedObject.ActualWidth, AssociatedObject.ActualHeight));
if (rect.Contains(pt))
{
if (!inBounds)
{
Debug.WriteLine("Here I am");
inBounds = true;
}
// Now, let's test to see if we are interested in this coordinate.
if (selectedRectangle == null || !selectedRectangle.Contains(pt))
{
GetElement(pt);
}
}
else
{
if (inBounds)
{
selectedTimer.Stop();
inBounds = false;
}
}
}, DispatcherPriority.Normal);
}
private Rect selectedRectangle;
private void GetElement(Point pt)
{
IInputElement element = AssociatedObject.InputHitTest(pt);
if (element == null) return;
TreeViewItem t = FindUpVisualTree<TreeViewItem>((DependencyObject)element);
if (t != null)
{
// Get the bounds of t.
if (selectedTreeview != t)
{
selectedTimer.Stop();
// This is a different item.
selectedTreeview = t;
selectedRectangle = selectedTreeview.RenderTransform.TransformBounds(
new Rect(0, 0, t.ActualWidth, t.ActualHeight));
selectedTimer.Start();
}
}
else
{
// Stop processing and drop this out.
selectedTimer.Stop();
selectedTreeview = null;
}
}
private T FindUpVisualTree<T>(DependencyObject initial) where T : DependencyObject
{
DependencyObject current = initial;
while (current != null && current.GetType() != typeof(T))
{
current = VisualTreeHelper.GetParent(current);
}
return current as T;
}
}
}

Of course, this is only useful if you have the Gesture handling code that I use as well. Being a nice guy, here it is:

using Goldlight.Perceptual.Sdk.Events;
using Goldlight.Windows8.Mvvm;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace Goldlight.Perceptual.Sdk
{
public class GesturePipeline : AsyncPipelineBase
{
private WeakEvent<GesturePositionEventArgs> gestureRaised = new WeakEvent<GesturePositionEventArgs>();
private WeakEvent<MultipleGestureEventArgs> multipleGesturesRaised = new WeakEvent<MultipleGestureEventArgs>();
private WeakEvent<GestureRecognizedEventArgs> gestureRecognised = new WeakEvent<GestureRecognizedEventArgs>();
public event EventHandler<GesturePositionEventArgs> HandMoved
{
add { gestureRaised.Add(value); }
remove { gestureRaised.Remove(value); }
}
public event EventHandler<MultipleGestureEventArgs> FingersMoved
{
add { multipleGesturesRaised.Add(value); }
remove { multipleGesturesRaised.Remove(value); }
}
public event EventHandler<GestureRecognizedEventArgs> GestureRecognized
{
add { gestureRecognised.Add(value); }
remove { gestureRecognised.Remove(value); }
}
public GesturePipeline()
{
EnableGesture();
searchPattern = GetSearchPattern();
}
private int ScreenWidth = 1024;
private int ScreenHeight = 960;
public void SetBounds(int width, int height)
{
this.ScreenWidth = width;
this.ScreenHeight = height;
}
public override void OnGestureSetup(ref PXCMGesture.ProfileInfo pinfo)
{
// Limit how close we have to get.
pinfo.activationDistance = 70;
base.OnGestureSetup(ref pinfo);
}
public override void OnGesture(ref PXCMGesture.Gesture gesture)
{
var handler = gestureRecognised;
if (handler != null)
{
handler.Invoke(new GestureRecognizedEventArgs(gesture.label.ToString()));
}
base.OnGesture(ref gesture);
}
public override bool OnNewFrame()
{
// We only query the gesture if we are connected. If not, we shouldn't
// attempt to query the gesture.
try
{
if (!IsDisconnected())
{
var gesture = QueryGesture();
PXCMGesture.GeoNode[] nodeData = new PXCMGesture.GeoNode[6];
PXCMGesture.GeoNode singleNode;
searchPattern = PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY;
var status = gesture.QueryNodeData(0, GetSearchPattern(), out singleNode);
if (status >= pxcmStatus.PXCM_STATUS_NO_ERROR)
{
var handler = gestureRaised;
if (handler != null)
{
handler.Invoke(new GesturePositionEventArgs(singleNode.positionImage.x - 85,
singleNode.positionImage.y - 75,
singleNode.body.ToString(),
ScreenWidth,
ScreenHeight));
}
}
}
Sleep(20);
}
catch (Exception ex)
{
// Ensure that we log the exception. We don't
// want to rethrow it.
Log(ex);
}
return true;
}
private static readonly object SyncLock = new object();
private void Sleep(int time)
{
lock (SyncLock)
{
Monitor.Wait(SyncLock, time);
}
}
private PXCMGesture.GeoNode.Label searchPattern;
private PXCMGesture.GeoNode.Label GetSearchPattern()
{
return PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY |
PXCMGesture.GeoNode.Label.LABEL_FINGER_INDEX |
PXCMGesture.GeoNode.Label.LABEL_FINGER_MIDDLE |
PXCMGesture.GeoNode.Label.LABEL_FINGER_PINKY |
PXCMGesture.GeoNode.Label.LABEL_FINGER_RING |
PXCMGesture.GeoNode.Label.LABEL_FINGER_THUMB |
PXCMGesture.GeoNode.Label.LABEL_HAND_FINGERTIP;
}
}
}

In order to use this, you’ll need my AsyncPipelineBase class:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace Goldlight.Perceptual.Sdk
{
public abstract class AsyncPipelineBase : UtilMPipeline
{
private readonly object SyncLock = new object();
public async void Run()
{
try
{
await Task.Run(() => {
while (true)
{
this.LoopFrames();
lock (SyncLock)
{
Monitor.Wait(SyncLock, 1000);
}
}
});
}
finally
{
this.Dispose();
}
}
}
}

If the gesture camera wasn’t hooked up to the computer, the call to LoopFrames drops out, which means that we have to maintain a loop for the lifetime of the application. I’m simply wrapping this in a while loop which sleeps for a second if the LoopFrames drops out. This seems to be a reasonable timeout for the application to wait until it picks up again.

Another big change for me this week was dropping the carousel. I know, it seems as though I’m chopping and changing the interface constantly, but that’s what happens in a development, you try something out, refine it and drop it if there’s a better way to do things. There were three main reasons for dropping the carousel:

1. Selecting a filter from elsewhere in the carousel also triggered the action associated with that filter. This could have been overcome, but the way it looked would have been compromised.
2. I put the carousel in place to give me a chance to use some of the pre-canned gestures. Unfortunately, I found using them to be far too cumbersome and “laggy”. I grew impatient waiting for it to cycle through the commands.
3. Using gestures with this just didn’t feel natural or intuitive. This was the biggest issue as far as I’m concerned.

Fortunately, the modular design of Huda allows me to take the items that were associated with the carousel and just drop them into another container with no effort. Two minutes and I had the filters in place in a different way. The downside is that Huda is becoming less and less like a Perceptual application as the interface favours touch more and more. Hmmmm, okay, I’ll revisit that statement. Huda feels less and less like an application that is trying too hard to be Perceptual. Over the last couple of weeks of development, I’ll be augmenting the touch interface with complementary gestures. For instance, I’d like to experiment with putting gesture pinch in place to trigger a x2 zoom. As the core infrastructure is in place, all I would really need to do here is hook the pinch recognition into the zoom command by passing a message. Now you see why I’ve gone with MVVM and message passing.

Part of the reason for the change in the interface came about because of something Steve talked about this week regarding “UserLaziness”. Quite naturally, the left and right hand sides of the screen relate to the operations that the user can perform to open photographs, and to manipulate the lists, but this just didn’t feel quite right. Basically, reordering the filters is something that I expect the user will do a lot less frequently than they will actually adding them, so it seems to me that we should reorder the interface a bit to move the filters to the bottom of the screen and to put the filter selection at the right hand side, where it hovers more naturally under the right hand thumb. This also means that the toolbar has become superfluous – again, it should be in the same location, minimizing the movements you should have to make.

One thing that is easy to get overlooked is that this is an Ultrabook application first and foremost. To that end, it really should make use of Ultrabook features. As a test, I hooked the Accelerometer up to the filters, so shaking it adds, appropriately enough, a blur filter.

This weeks music:

• AC/DC – For Those About To Rock
• AC/DC – Back In Black
• Bon Jovi – What About Now
• Eric Johnson – Ah Via Musicom
• Wolfmother – Cosmic Egg
• Airbourne – Runnin Wild
• DeeExpus – The King Of Number 33
• Dream Theater – A Dramatic Turn Of Events

Status Update

So where are we now? Well, Huda feels like an Ultrabook app to me first and foremost. When I start it up, I don’t really need to use the mouse or keyboard at all – the desktop side of it isn’t as necessary because the Ultrabook really makes it easier for the user to use touch to accomplish pretty much everything. I’m not too happy with the pinch zoom and touch drag though – it does very, very odd things – I’ll sort that out so that it is much closer to what you’d expect with a photo editing package.

One point of design that I’d like to touch upon. Nicole raised an interesting point about not saving the filtered version of the photo into the same location as the original. My intention was always to have a separate Export directory in which to store the images, but I still haven’t decided where that’s going to be. So Nicole – here’s your chance to really shape an application to do what you want – where would you like to see exported images reside?

Finally. While playing around with adding filters, it quickly becomes apparent that it’s possible to add them far quicker than they can be applied and rendered. Now, there are things I can do to help with this, but I am making the tough call here that the most important thing for me to concentrate on right now is adding more gestures, and making sure that it’s as robust and tested as I can possibly make it. If there’s time, and it doesn’t impact on the stability, I will mitigate the filter application. The issue is to do with the fact that Bitmap and BitmapSource conversions happen in a lot of the filters. As the filters are reapplied to the base image every time, this conversion could happen each time – I could cache earlier image conversions and just apply the new filter, but reordering the filters will still incur this overhead. As I say, I’ll assess how best to fit this in.

Well, that’s me for this week. Thanks for reading and if there’s anything you need to know, please let me know.

Ultimate Coder – Ma il Mio Mistero è Chiuso In Me

This is a copy of the post I made on the Intel site here. For the duration of the contest, I am posting a weekly blog digest of my progress with using the Perceptual Computing items. This weeks post shows how Huda has evolved from the application that was created at the end of the third week. My fellow competitors are planning to build some amazing applications, so I’d suggest that you head on over to the main Ultimate Coder site and follow along with their progress as well. I’d like to take this opportunity to wish them good luck.

Executive summary

The judges will be pleased to know that this blog entry is a lot less verbose than previous ones. This isn’t because there’s not a lot to talk about, but rather because I’m concentrating on writing a lot of code this week, and I’ve hit a real roadblock with regards to the gesture camera. Code that was working perfectly well last week has suddenly stopped working. This means that I’ve got some serious debugging to do here to find out why the gesture code is no longer displaying the cursor on the screen – this doesn’t happen all the time, but it’s happened often enough for me to be concerned that there’s an underlying problem there that needs to be sorted out.

As part of my remit is to help coders learn to write code for the Perceptual SDK, I will be continuing writing about the coding process, so I apologise to the judges in advance – there’s nothing I can do to alleviate the detail of explanation. If they wish to skip ahead, the summary talks about where we are today and what my plans are for next week, and here are some screen shots. Unlike my previous blogs, I’m not going to be breaking this one down on a day by day basis. There’s a lot that needs to be done, and it doesn’t break down neatly into days.

For those who want to know the result, don’t look away now. I found and fixed the issue with the gesture code. It’s all a matter of patient debugging and narrowing down areas where there can possibly be problems.

I’ve been wondering how best to train people in how to use Huda, and this week I’ve hit on the solution. I’m going to let Huda teach you. By the end of the week, I added in voice synthesis, and I think I can use this to help provide context. The type of thing I have in mind is, say you bring up the filters, Huda will tell you that a swipe left or right will cycle through the filters, and the thumbs up will add the filter to the currently selected image. The beauty is, now that I have the code in place to do the voice synthesis, I can easily add this type of context.

Voice recognition is an area of the Perceptual SDK that has a long way to go. It just isn’t accurate enough and, for that reason, I’m dropping support for it from Huda. I had the commands “Open Filter” and “Close Filter” in the application, and no matter what I tried, it kept turning Open Filter into Close Filter. Yes, it recognises a few words, but given that the accuracy is so hit and miss, I can’t leave it in there. So, I apologise to you right now – voice control is out. I had to take this decision this week because there are certain things I was using it for that I have to find other ways to accomplish these things and I really don’t have the time to keep mucking around with this.

If you don’t want to read the rest of the post, this is what Huda looks like now:

Woe to the interface

It’s going to be a busy week this week. The more I play with the interface, the more I find it clunky and unpolished, and this is incredibly frustrating. While I know that I’m a long way from polishing the interface, I can’t help but think that I’m missing a trick or two here. The biggest thing that I have issues with is the concept of showing the folders and selecting the images. So, the first thing I’m going to do is to unify the two parts; this should make it obvious that the picture selection and folder selection actually belong together. Part of my thinking here is that I want it to be apparent to anyone who’s used a photo editing application before that this is still a photo editing application. While it’s fine to play about with an interface, there still has to be a certain level of familiarity; there has to be a feeling of comfort in using the application otherwise the user will end up being overwhelmed and will give up on the application. While the contest is about working with Perceptual computing and features of the ultrabook, it’s important that I don’t lose sight of the fact that this must be an application that people can use.

Once I’ve got this in place, I want to address the selection of “things to do to the photograph”. By this, I mean that I want to put some form of toolbar into place to trigger options. Don’t worry that I’m throwing sections of the UI together – I’m not. What I will be doing here is giving the user obvious trigger points – I can use this to bridge the gap between traditional photo editing applications and Huda. There has to be some familiarity for the user here, so I’ll be providing that today.

You may think that this is going to take a lot of work, and in other languages you may well be right, but I’m using MVVM and WPF. As the different parts of the interface are loosely coupled controls, it’s a simple matter for me to move the controls into other containers, and they will still work. The only change I’m going to make is to keep the images showing regardless of whether or not the user has opened a folder.

With the basics of the “toolbar” in place, I now have something that feels more natural for using gestures with. I’m going to hook the filter selection into both touch and gestures, and do something a little bit fancy for displaying the list of filters. What I’ll do is move the filters into a carousel, and use the swipe left/swipe right to move backwards and forwards through the list.

I’m concerned about the speed of loading of various items when thingsare selected, so I will be moving things out onto background threads.

The interface is really starting to firm up now. It’s starting to feel that little bit more polished to me, but there is more that I can do with it.

Don’t worry that all the filters have the same name and appearance right now. This is purely a visual that will be changed soon – I need to come up with some good iconography for these. It’s strangely satisfying swiping your hand left and right to rotate through this list. My final action on this end will be to introduce a gesture that actually applies the filter. The thumbs up seems oddly appropriate to me.

The big reveal

After a frenzy of activity on the actual interface, I want to address one of the central concepts of Huda; namely, the ability to filter the image, and not touch the underlying image. This is where I give up the secret of Huda – it’s all a massive con. If you remember, I promised that the underlying image would not be touched. I’ve had people get in touch with all sorts of wild theories on how I was going to do this – from the person who thought I was going to use steganography to hide this information in the picture to someone who thought I was going to convert the file into a custom format and put something in place to fool image editors into thinking that the type was still the original file format.

If I were to tell you that the solution to this problem was incredibly straightforward, and blindingly obvious, would this surprise you? Well, here it is. Here’s how it’s done…. Did you catch that? Yes, it’s all smoke and mirrors. What Huda does is actually not modify the original image at all – instead, we maintain an index of all the files that we’ve edited in Huda, along with the details of all the edits that have been performed, and these can be played again and again and again. In other words, when you open a file up in Huda, it goes to the index first to see if you’ve previously opened it. If you have, it gets the image transformations and reapplies them there and then. It’s as simple as that. Of course, this does rely on you not moving photos around, but I do have some plans in place for a post contest version of Huda to take care of that as well. Anyone for hashing?

Transformers – Robert’s in disguise

A judgely warning – this section is going to be intense. I’m sorry, but there’s no way around the fact that we need to write some code, and there needs to be some explanation of it. It’s going to be quite dry, so I won’t blame you if you want to skip to the bit at the end.

As a first pass for the transformations that I want to apply, I decided that I needed to declare a common interface that all my transformation classes would apply. It’s a very simple interface, but as I want all my transformations to be serializable, I’ve made sure that it implements ISerializable by default

using System.Windows.Media.Imaging;
using System.Runtime.Serialization;
namespace Huda.Transformations
{
    public interface IImageTransform : ISerializable
    {
        int Version { get; }
        BitmapSource Transform(BitmapSource source);
    }
}

At this point it’s merely a matter of creating some simple transformations that use this functionality. Being the nice chap that I am, and to give Nicole some coding kudos that she can drop into casual conversation, I’ll show you all the classes I’m using for transformation. First of all, here’s the image crop routine:

using System.Windows.Media.Imaging;
using System.Windows;
using System.Runtime.Serialization;
using System;
namespace Huda.Transformations
{
    [Serializable]
    public class CropImage : IImageTransform
    {
        public CropImage()
        {
            Version = 1;
        }
        protected CropImage(SerializationInfo info, StreamingContext context) : this()
        {
            Version = info.GetInt32(Constants.Version);
            Left = info.GetInt32(Constants.Left);
            Top = info.GetInt32(Constants.Top);
            Width = info.GetInt32(Constants.Width);
            Height = info.GetInt32(Constants.Height);
        }
        public int Version
        {
            get;
            private set;
        }
        public int Left { get; set; }
        public int Top { get; set; }
        public int Width { get; set; }
        public int Height { get; set; }
        public BitmapSource Transform(BitmapSource source)
        {
            Int32Rect rect = new Int32Rect(Left, Top, Width, Height);
            source = new CroppedBitmap(source, rect);
            return source;
        }
        public void GetObjectData(SerializationInfo info, StreamingContext context)
        {
            info.AddValue(Constants.Version, Version);
            info.AddValue(Constants.Left, Left);
            info.AddValue(Constants.Top, Top);
            info.AddValue(Constants.Width, Width);
            info.AddValue(Constants.Height, Height);
        }
    }
}

There’s a lot of code in here that you’ll see in other classes, so I’ll just explain it once and you should easily be able to follow how I’m using it in other locations. The first thing to notice is that I’ve marked this class as Serializable – you need to do that if you want .NET to do it’s mojo when you save things. You’ll see that there are two things inside the class that say CropImage; these are the constructors that are used to create the object. The second one is the special constructor, and is the reason that I had IImageTransform implemenent ISerializable. .NET knows, when it sees this interface, that you want to read and write out the values in the class yourself. There are many, many reasons that I want to do this, but the main reason is because of what happens if you just let the runtime try and figure this stuff out itself – it’s not very performant because of the way it has to map things together. By taking control of this myself, I make the job easy. If you see how I use info.GetInt32 (there are many other methods for reading other types of data), it’s apparent what type of data the property is – we must always make sure we get it right, otherwise unpredictable and nasty things can happen.

At the bottom of the code, there’s a matching method called GetObjectData that just writes the items to the serialization stream. It’s important to make sure that you use the same names for the items in the constructor that you did in this method. The property names don’t have to match these names, but the names themselves do.

The Version currently does nothing, but if I have to add features to any of these classes, I can use this to work out which properties should be present, so the saving and reading of the data carries on seamlessly. It’s always a good idea to design your serialization with this firmly in mind.

The Left, Top, Width and Height properties must be set before we attempt to call Transform. If we forget to set them, all that will happen is that the rectangle that’s created will be 0 pixels in size, and we wouldn’t want a zero pixel crop.

The Transform method is where the “clever” stuff actually happens. I bet you thought there would be more to it. All it does is create a rectangle based on the size and location set in the properties, and then it uses the inbuilt CroppedBitmap class to actually perform the crop.

See! I told you that was easy. Next, I’ll show you what the FlipImage transformation looks like:

using System;
using System.Windows.Media.Imaging;
using System.Windows.Media;
using System.Runtime.Serialization;
namespace Huda.Transformations
{
    [Serializable]
    public class FlipImage : IImageTransform
    {
        public FlipImage()
        {
            Version = 1;
        }
        protected FlipImage(SerializationInfo info, StreamingContext context)
        {
            Version = info.GetInt32(Constants.Version);
            HorizontalFlip = info.GetBoolean(Constants.Horizontal);
        }
        public int Version { get; private set; }
        public bool HorizontalFlip { get; set; }
        public BitmapSource Transform(BitmapSource source)
        {
            source = new TransformedBitmap(source, new ScaleTransform(HorizontalFlip ? -1 : 1, HorizontalFlip ? 1 : -1));
            return source;
        }
        public void GetObjectData(SerializationInfo info, StreamingContext context)
        {
            info.AddValue(Constants.Version, Version);
            info.AddValue(Constants.Horizontal, HorizontalFlip);
        }
    }
}

I won’t explain this code, as it’s largely the same as the CropImage code. If you’re interested why I didn’t put things like the Version and Transform into an abstract base class, please let me know and I’ll cover attempt to answer it in the comments. At this point, the judges are screaming for mercy and begging me to stop, so I’ll try not go too far into philosophical architectural debates.

The ResizeImage transformation is an interesting one (well, interesting if you’re into that type of thing), and the Transform code is more complex than other tranformations. To save you having to fight with this yourself, here it is

using System;
using System.Windows.Media.Imaging;
using System.Windows.Media;
using System.Windows;
using System.Runtime.Serialization;
namespace Huda.Transformations
{
    [Serializable]
    public class ResizeImage : IImageTransform
    {
        public ResizeImage()
        {
            Version = 1;
        }
        protected ResizeImage(SerializationInfo info, StreamingContext context)
            : this()
        {
            Version = info.GetInt32(Constants.Version);
            Width = info.GetInt32(Constants.Width);
            Height = info.GetInt32(Constants.Height);
        }
        public int Version
        {
            get;
            private set;
        }
        public int Width { get; set; }
        public int Height { get; set; }
        public BitmapSource Transform(BitmapSource source)
        {
            Rect rect = new Rect(0, 0, Width, Height);
            DrawingVisual visual = new DrawingVisual();
            using (DrawingContext ctx = visual.RenderOpen())
            {
                ctx.DrawImage(source, rect);
            }
            RenderTargetBitmap resized = new RenderTargetBitmap(
                Width, Height, 96, 96, PixelFormats.Default
                );
            resized.Render(visual);
            return resized;
        }
        public void GetObjectData(SerializationInfo info, StreamingContext context)
        {
            info.AddValue(Constants.Version, Version);
            info.AddValue(Constants.Width, Width);
            info.AddValue(Constants.Height, Height);
        }
    }
}

The constants we’d use

namespace Huda.Transformations
{
    public class Constants
    {
        public const string Left = "Left";
        public const string Right = "Right";
        public const string Top = "Top";
        public const string Bottom = "Bottom";
        public const string Width = "Width";
        public const string Height = "Height";
        public const string Horizontal = "Horizontal";
        public const string Version = "Version";
    }
}

I have been asked how the gesture hookup behaves. Well, imagine you were tracking mouse moves to see which items were under the cursor, you might want to create a Blend Behavior that looked something like this:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Interactivity;
using System.Windows.Input;
using System.Windows.Controls;
using System.Diagnostics;
using System.Timers;
using System.Windows.Media;
using LinqToVisualTree;
using System.Windows.Threading;

namespace Huda.Behaviors
{
    public class GestureMovementBehavior : Behavior<TreeView>
    {
        private bool inBounds;
        private Timer selectedTimer = new Timer();
        private TreeViewItem selectedTreeview;
        private TreeViewItem previousTreeView;

        protected override void OnAttached()
        {
            Window.GetWindow(AssociatedObject).MouseMove += GestureMovementBehavior_MouseMove;
            selectedTimer.Interval = 2000;
            selectedTimer.AutoReset = true;
            selectedTimer.Elapsed += selectedTimer_Elapsed;
            base.OnAttached();
        }

        void selectedTimer_Elapsed(object sender, ElapsedEventArgs e)
        {
            selectedTimer.Stop();
            Dispatcher.Invoke(DispatcherPriority.Normal,
                (Action)delegate()
                {
                    if (previousTreeView != null)
                    {
                        previousTreeView.IsSelected = false;
                    }
                    previousTreeView = selectedTreeview;
                    selectedTreeview.IsSelected = true;
                });
        }

        protected override void OnDetaching()
        {
            Window.GetWindow(AssociatedObject).MouseMove -= GestureMovementBehavior_MouseMove;
            base.OnDetaching();
        }

        void GestureMovementBehavior_MouseMove(object sender, MouseEventArgs e)
        {
            // Are we over the treeview?
            Point pt = e.GetPosition(Window.GetWindow(AssociatedObject));
            Rect rect = new Rect();
            rect = AssociatedObject.RenderTransform.TransformBounds(
                new Rect(0, 0, AssociatedObject.ActualWidth, AssociatedObject.ActualHeight));
            if (rect.Contains(pt))
            {
                if (!inBounds)
                {
                    inBounds = true;
                }
                // Now, let's test to see if we are interested in this coordinate.
                if (selectedRectangle == null || !selectedRectangle.Contains(pt))
                {
                    GetElement(pt);
                }
            }
            else
            {
                if (inBounds)
                {
                    selectedTimer.Stop();
                    inBounds = false;
                }
            }
        }

        private Rect selectedRectangle;
        private void GetElement(Point pt)
        {
            IInputElement element = AssociatedObject.InputHitTest(pt);
            if (element == null) return;

            TreeViewItem t = FindUpVisualTree<TreeViewItem>((DependencyObject)element);
            if (t != null)
            {
                // Get the bounds of t.
                if (selectedTreeview != t)
                {
                    selectedTimer.Stop();
                    // This is a different item.
                    selectedTreeview = t;
                    selectedRectangle = selectedTreeview.RenderTransform.TransformBounds(
                        new Rect(0, 0, t.ActualWidth, t.ActualHeight));
                    selectedTimer.Start();
                }
            }
            else
            {
                // Stop processing and drop this out.
                selectedTimer.Stop();
                selectedTreeview = null;
            }
        }

        private T FindUpVisualTree<T>(DependencyObject initial) where T : DependencyObject
        {
            DependencyObject current = initial;

            while (current != null && current.GetType() != typeof(T))
            {
                current = VisualTreeHelper.GetParent(current);
            }
            return current as T;
        }
    }
}

With only a minimal amount of change, that’s exactly the Behavior that you hook up to a treeview.

 

And there you have it – code to do some basic transformations. The other filters that I apply will all follow this pattern, so it’s an incredibly trivial thing for me to put in place – and, more importantly, this provides the core functionality for me to save out the filters along with details of the files they apply to. All of a sudden, this application is starting to come together.

This weeks music

• Deep Purple – Purpendicular
• Elbow – Build A Rocket Boys
• David Garrett – Rock Symphonies
• Crash Test Dummies – God Shuffled His Feet
• Nickelback – All The Right Reasons
• Jethro Tull – Aqualung Live
• Creedance Clearwater Revival – Creedance Clearwater Revival
• Deep Purple – The Battle Rages On
• Caitlin – The Show Must Go On

Status update

I’ve reached the end of the week and it’s worth taking stock of where I’ve reached. The harsh reality right now is that I won’t be delivering a fully-featured photo editing application at the end. Before the alarm bells start ringing, this is exactly what I expected. The aim for this competition is not to release the next Photoshop at the end of week 7. This would be a physical impossibility, given how I’m incorporating none Photoshop features. I am confident, however, that I will deliver what I was aiming at from the very start – a none destructive gesture and touch driven photo editing application. For this reason, I am going to only going to incorporate a limited number of filters. Don’t worry though, if you want to use Huda, I will continue developing it once the contest has finished. Your investment in using it will not be wasted.

Right now, Huda is limited to only editing a single image at a time. If I have time, before the end of the contest, I will give Huda the ability to edit multiple photos. Over the next week, I’m going to be tackling the reordering of items in the list, fleshing out that toolbar and also having the interface behave a little bit differently when it’s in tablet or desktop mode. Right now, the design has all been based on a tablet experience – but let’s see what we can do for the desktop users. Oh, and I should also have the first draft of the voice context in place as well.