Summary

Hyper-video is a concept that a number of people have been thinking about for quite a while, though it is entirely possible that we each have been using the term to mean something slightly different.

For the purposes of this brief tutorial, hyper-video will be defined to be a hyphenated term meaning the ability to make an intuitive gesture (such as clicking on a video player) that indicates you wish to pause a video in order to obtain more information (typically by opening a related video), much as hyper-text allows you to click on a block of text to link to a new article.

The concept of HyperVideo is most easily explained by a series of examples:

• You are watching the news and the anchor person says "Robert Mugabe, president of Zimbabwe, today announced…" you click on the question mark on 2/5/2009 the screen and are presented with a menu:

      1. Zimbabwe History (Video)
      2. Zimbabwe Government and Politics (Video)
      3. President Mugabe Biography (Video)
      4. Zimbabwe article on Wikepedia

• You are watching a video on how to use a Silverlight Toolkit control and you realize that you don't know how to install the Toolkit. You click on the glowing question help button and the video pauses and a second video opens that provides the background information you need.
• You are watching an HR training video and the person in the video mentions investing in a 401K. The word 401K appears on the screen and when you click on it, the first video pauses and a second video begins detailing your retirement plan options. What you don't necessarily realize, however, is that the supplemental video you are seeing is targeted at your insurance options; your co-worker might see a different video.

Building a Hyper Video Player

Hyper-video is created by the cooperation of three components:

• One or more videos
• Markers embedded into the video that are invisible to the user but that are perceived by the player
• The hyper-video player that detects the marker and handles the user's "gesture" indicating a wish to follow the "link."

The Tools We'll Use

While there are many ways to accomplish this work, we'll make our lives easy by using three tools that will greatly simplify the work ahead:

• Visual Studio 2008
• Expression Blend
• Expression Encoder

Expression Encoder

Of the three, the one you probably don't have yet is Expression Encoder

If you've not yet worked with Blend or Visual Studio 2008, this is probably the wrong tutorial for you right now. You'll want to begin by working your way through some of the earlier tutorials and then come back for this relatively advanced material.

To get Encoder, navigate to the Expression Web Site.

Figure 10-1. Expression Web Site (Click to view full-size image)

In the lower right hand corner click on Encoder

Figure 10-2. Try or Buy Encoder (Click to view full-size image)

Download (or buy) Encoder 2, and be sure to upgrade to SP1.

Once you are set up, you are ready to use Encoder to add markers to your video and to ask Encoder to emit a player for you that you will modify, which is infinitely easy that writing the player from scratch.

The Video

You'll need a video to start your work. You can use any video you like; I recommend downloading a wmv version of any of the How Do I videos available on our site; something relatively short is ideal such as this video on Databinding.

Importing the Video

Open Encoder and choose File->Import

Figure 10-3. Importing the video

Navigate to the video you want to import. It will appear in the Media content window of Encoder and you can watch it by pressing the play button.

Figure 10-4. Marker Timeline (Click to view full-size image)

The area I marked A represents the imported video. On the control toolbar you can see the elapsed time (area B). Above that, to the left of the indicator marked C is the orange playhead.

MetaData

The right side of Encoder has four tabs:

• Encode
• Enhance
• MetaData
• Output

For now, we care only about the last two. The Metadata tab offers the ability to name the output video (and add an enormous amount of metadata such as copyright, genre, etc.) and to add markers,

Figure 10-5. Adding First Marker

There are many ways to add markers, but one of the easiest is to move the playhead to the place in the video you want the marker, and then to press the Add button.

Four editable fields are made available for each marker:

• Time (Elapsed time into the video in thousandths of a second)
• Value (any arbitrary value that will serve as a name for the marker)
• Thumbnail (a boolean indicating whether or not a thumbnail should be created)
• Keyframe (a boolean indicating whether or not a keyframe should be created)

Figure 10-6. Four editable fields

The thumbnail is used when creating chapters (as for a DVD movie) to show an image indicating what is in that scene of the film.

Creating a keyframe instructs Encoder to put an IFrame at that location. The documentation notes that this can shorten seek time for the marker" but again is not relevant for the work we'll be doing.

For our purposes Thumbnail and Keyframe should be unchecked.

What to Put In Value

Traditionally, the Value field is filled with some indication of what is happening in the video at that time location, though see my previous blog entry for a discussion of why that may or may not be desirable.

Once You've Added the Markers

Once you've added all the markers that you wish to add, and adjusted their exact timing to whatever level of precision you choose, you are ready to encode the video and to enlist Encoder in creating a player for you.

Figure 10-7. Markers on the time line and the marker list (Click to view full-size image)

Creating the Player

Switch to the output panel, where you will pick a Silverlight 2 player and then click on the white dot (shown surrounded in red here)

Figure 10-8. SettingTheTemplate

When you choose the Silverlight 2 Default player template an image of the player will appear.

Figure 10-9. EditTemplate Preview

Clicking on the dot will open a menu asking how you want to edit the template.

Figure 10-10. TemplateMenu

Choosing Edit Copy in Expression Blend will open a dialog box asking you for a name for the template. Give it a name and click OK and Blend will open with a security warning. You can click Yes and Blend will fully open and inform you that you must build the project to work with it. Click Project->Build Solution and when it completes you should see the player in the art board and the Objects and Timeline will indicate that within the Layout Root is a single object: myPlayer.

Figure 10-11. BlendWithMyPlayer (Click to view full-size image)

Encoding the Video

Before you go any further, don't forget to finish encoding the video! To do so, return to Encoder, make sure the output tab is active, and examine the Media File Name and the Directory,

Figure 10-12. EncoderDirectory

The default name for your new encoded video is created by appending the default extension (wmv) to the original file name. I've added the text "-Encoded" to remind myself that this is the encoded version.

You can also pick the directory you'd like to place the encoded video into. There are some checkboxes below that can be helpful as well.

Once you have this set as you like, click the Encode button in the lower left hand corner and your video will begin encoding, with a countdown clock telling you how much time remains. If you checked "Preview in Browser" when you finish a browser will open and your video will display (though there may be a short delay).

Files, Files, Who Has The Files?

At this point you have two sets of important files:

• The encoded video and some helper files to look at it
• The emitted player and the source code for modifying it

The encoded video should be in the directory you chose in the example shown above, the files would be in c:\demo\HyperVideo Tutorial\ \VISTA64DT 2-29-2008 12.45.51 PM\ where VISTA64DT is the name of my machine and the rest of the name is a time stamp.

Inside that directory will be a copy of the .xap file for the emitted player, along with a .dat file that contains a great deal of information about the video and the markers, a default.html and the encoded wmv file. Double clicking on the html file will bring up your video in the emitted video player.

Where is the Player?

The player itself is in the default output location unless you took steps to put it elsewhere. On my machine that is C:\Users\Jesse\Documents\Expression\Expression Encoder\Templates.

Under that directory I will find a directory with the name of the template, in this case, Silverlight 2 Default Tutorial inside of which is another Default.html and copy of the .xap file, but more important, is also a Source directory. In that directory is the complete source (and Visual Studio solution files!) for the custom control player that I will want to modify.

Chapters

When the video begins playing click on the button in the lower right that I've highlighted and pointed to in figure 10-13. That causes the "chapters" to be revealed, a feature we won't use, but you can see that the markers have each established a chapter and that the time and name of the marker is shown. Had you added keyframes, there would be a corresponding image as well.

Figure 10-13. FullPlayerRunning. (Click to view full-size image)

Spec'ing the Player

While this player looks great, and is terrifically enhanced by the fact that we did no work to create it, we really don't want the user to open up "chapters" as that is not how we're using our markers. We added the markers as metadata that we will respond to as if they were links to other related data outside of this video – same syntax but an entirely different set of semantics.

Thus, we want to remove the chapters button, and the video list button to its left and the full screen button to its right and also the closed caption button to the far right.

While we're simplifying, we'll also eliminate the buttons to skip ahead and back a chapter (We ain't got no chapters. We don't need no chapters! I don't have to show you any stinkin' chapters! ^[1]).

Therefore, we will delete the eight buttons highlighted in yellow in figure 10-14

Figure 10-14. WhatToDelete (Click to view full-size image)

You can see, however, why we wanted Encoder to create the player; it is far easier to start with this and remove what we don't want than to create all the player functionality from scratch.

Working With the Project

Return to your code folder, and open the solution in both Blend and Visual Studio

C:\Users\Jesse\Documents\Expression\Expression Encoder\Templates\Silverlight 2 Default Tutorial\Source\Template.sln

Let's start in Visual Studio, in Page.xaml and identify all the code for the video player and each of the buttons.

<usercontrol x:Class="MediaPlayerTemplate.Page" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:ExpressionPlayer="clr-namespace:ExpressionMediaPlayer;assembly=ExpressionPlayer" mc:Ignorable="d" Width="Auto" Height="Auto"> <grid x:Name="LayoutRoot" /> <ExpressionPlayer:ExpressionPlayer x:Name="myPlayer" Margin="0,0,0,0" />

We had hoped to find something like this

<rectangle x:Name="player" Width="150" Height="200" /> <button type="button" x:Name="SkipBackOneChapter" Style="{StaticResource PlayerButton}"></button> <button type="button" x:Name="Play" Style="{StaticResource PlayerButton}"></button> <button type="button" x:Name="SkipForwardOneChapter" Style="{StaticResource PlayerButton}"></button> <button type="button" x:Name="ShowChapters" Style="{StaticResource PlayerButton}"></button>

We could then just cut the buttons we didn't want. Instead all we have is a single, seemingly indivisible control of type ExpressionPlayer.

It's Just A Skinnable Custom Control

What Encoder has actually provided to you is a skinnable custom control, exactly as described in these four videos:

• Building a skinnable custom control part 1
• Building a skinnable custom control part 2
• Building a skinnable custom control part 3
• Building a skinnable custom control part 4

In fact, Encoder has created a custom class, ExpressionPlayer that derives from MediaPlayer.

public class ExpressionPlayer : MediaPlayer

ExpressionPlayer is not given a default look, but inherits its appearance from MediaPlayer (we see that same trick in my Carousel Video where the CarouselVideo inherits its appearance from Panel).

Fortunately, however, MediaPlayer's source is provided, and as you'd expect there is a Themes folder with a generic.xaml file that does contain the default appearance.

In addition, MediaPlayer.cs provides the attributes that constitute the Parts and States contract.

For more on the Parts and States contract and how attributes are used in this context, please see Building a Skinnable custom Control Part 1 and Part 2.

Here is an excerpt:

[TemplatePart(Name=MediaPlayer.StretchBox, Type=typeof(FrameworkElement))] [TemplatePart(Name=MediaPlayer.VideoWindow, Type=typeof(FrameworkElement))] [TemplatePart(Name=MediaPlayer.TextblockErrorMessage, Type = typeof(TextBlock))] [TemplatePart(Name=MediaPlayer.ClosedCaptionBackground, Type = typeof(Rectangle))]

All of this makes skinning (templating) the Expression player far easier.

Templating the Player

To begin templating the player, right click on the player control and choose Edit Control Parts. Be sure to choose Edit a Copy rather than Create Empty as we'll want to keep most of the player, just chipping away those parts we don't need.

Figure 10-15. Create a template for the player control (Click to view full-size image)

You are prompted, as usual to pick a name and to choose whether to put the template into the file or into App.xaml (for the entire application). We'll choose the latter.

Once these choices are made you are dropped into the template editor, where the Parts and States model is made manifest in the States tab and the Objects and Timeline tab respectively.

Figure 10-16. Player in Template Editor (Click to view full-size image)

How do you sculpt an Elephant? ^[2]

Expand all the parts revealed in the objects and Timeline to see all the sub-parts and drill down into the player components. Delete the controls that are not needed for the hyper-video player.

A good place to start is to delete the set of "miscControls" Notice that when you click on the "miscControls" in the objects and timeline tab, those controls are conveniently highlighted on the player to indicate which controls you are about to assassinate,

Figure 10-17. Deleting "miscControls" (Click to view full-size image)

By clicking delete on the "miscControls" container you delete all the contained controls at one go.

Figure 10-18. Removing Controls (Click to view full-size image)

In the same way, you can go on to eliminate the buttons used for rapid navigation (step backwards, step forwards, buttonPrevious and buttonNext). You can highlight more than one control using control-click, and delete them all at the same time. Be sure to laugh maniacally as you do so.

Figure 10-19. Multi-Delete

This will strip the player down to the essentials.

Figure 10-20. Player Essentials (Click to view full-size image)

We're now ready to feed our video, complete with its embedded markers, to our templatized player.

Responding to the Markers

I'd like to tell you that responding to the markers that we added to the video involves a bit of tricky programming, and you're lucky you found this tutorial because I can show you how to do it right… I'd like to tell you that, but actually, like virtually everything involved in this process, it is very straight forward.

Save but do not close the project in Blend, and open it in Visual Studio.

You'll remember that what we've created is a template for the ExpressionPlayer class that was emitted by Encoder and that is found in ExpressionPlayerClass.cs:

namespace ExpressionMediaPlayer { public class ExpressionPlayer : MediaPlayer

Stripping out the using statements and comments reveals that the class created by Encoder is just a specialization (derived class) of MediaPlayer.

It turns out that MediaPlayer's source is provided as well. Open MediaPlayer.cs and scroll down to the Events region, where you will find the following event declaration:

public event TimelineMarkerRoutedEventHandler MarkerReached;

That is the event we want to respond to; it is fired every time the player encounters a marker in the video.

As with all events, your handler will receive two parameters: a reference to the object that fired the event (the player) and an object that is or derives from EventARrgs.

In this case, the second parameter is of type TimelineMarkerRoutedEventArgs, which is well documented in the Silvelright documentation and which includes (among other things) a Marker property. That Marker has, in turn, three important properties

Figure 10-21. Marker Properties (Click to view full-size image)

The TimelineMarker type is the meta-data in the video, and Text, Time and Type provide exactly the information you need for your hyper-video.

The EventArgs type that is provided is well documented in the standard Silverlight documentation, which includes among other things sample code that makes our work embarrassingly easy,

public void OnMarkerReached(object sender, TimelineMarkerRoutedEventArgs e) { timeTextBlock.Text = e.Marker.Time.Seconds.ToString(); typeTextBlock.Text = e.Marker.Type.ToString(); valueTextBlock.Text = e.Marker.Text.ToString(); }

All we need is a TextBlock to display the values we'll receive when we hit the marker. Open Page.xaml and replace the grid with the following code (which adds two rows to the grid, and a TextBlock in the second row),

<grid x:Name="LayoutRoot" /> <grid.rowdefinitions /> <rowdefinition Height="9*" /> <rowdefinition Height="0.5*" /> <ExpressionPlayer:ExpressionPlayer Margin="0,0,0,0" x:Name="myPlayer" Style='{StaticResource ExpressionPlayerBlogVersion }' Grid.Row='0' /> <textblock x:Name='Message' Grid.Row='1' />

You can now implement the event handler in Page.xaml.cs,

public partial class Page : UserControl { public Page( object sender, StartupEventArgs e ) { InitializeComponent(); myPlayer.OnStartup( sender, e ); myPlayer.MarkerReached += new TimelineMarkerRoutedEventHandler( myPlayer_MarkerReached ); } void myPlayer_MarkerReached( object sender, TimelineMarkerRoutedEventArgs e ) { Message.Text = e.Marker.Text + " at " + e.Marker.Time.Seconds.ToString() + " seconds."; } }

The easiest way to whether this is working this is to build the project and then to navigate to the output directory you designated in encoder, and pick up a copy of the Default.html and the .wmv file that was output for you. Drop these in the debug directory of your project folder.

Figure 10-22. Copy Files (Click to view full-size image)

Double click on Default.html and "let 'er rip!"

Figure 10-23. Showing the Marker (Click to view full-size image)

You can see the marker indicated just below the (cropped) player.

Note carefully, each time you recompile, the default.html in the source project will be over-written, which is why you'll want to copy default.html from your working directory.

From Here to Hyper-video

We now have all the components, we need only finish up by putting in the logic to show an actual video rather than just the marker title when we hit each marker. To do so, we'll create clips for display and place them in a clips directory (though in a deployed program you will probably want them in a more extensible location such as a database).

Figure 10-24. Clips (Click to view full-size image)

Adding a Media Player

Each of the clips corresponds to one of the markers in the encoded video. Our next step is to update the Xaml to replace the text box with a Media player to show these clips,

<grid x:Name="LayoutRoot" /> <grid.rowdefinitions /> <rowdefinition Height="9*" /> <rowdefinition Height="2*" /> <rowdefinition Height="7*" /> <ExpressionPlayer:ExpressionPlayer Margin="0,0,0,0">x:Name="myPlayer" Style="{StaticResource myPlayer}" Grid.Row="0" /> <mediaelement x:Name="VideoLink" />Visibility="Collapsed" Grid.Row="2" /> <button type="button" Height="40">HorizontalAlignment="Left" VerticalAlignment="Bottom" Width="110" Grid.Row="1" Content="Show More!" x:Name="ShowMore" FontFamily="Georgia" FontSize="18" Visibility="Collapsed" />

Notice that both the MediaElement and the Button start out with their visibility set to collapsed, We'll display the button at each marker (for a set amount of time) and if the user clicks on the button (requesting more information) we'll pause the main video and display the MediaElement with the appropriate supporting video.

All of this logic is handled in the code-behind for Page.xaml.

To begin, we'll remove the contents of the MarkerReached method and replace with new logic. The first step is to create a Timer. This requires that we add a using statement,

using System.Threading;

With that in place we can add the call to the Timer's constructor,

Timer t = new Timer( EndShowMore, // call back ShowMore, // state 2000, // dueTime System.Threading.Timeout.Infinite // period );

These fields are documented in the standard documentation for System.Timer,

Figure 10-25. Timer Constructor Documentation (Click to view full-size image)

You'll need the TimelineMarkerRoutedEventArgs when the timer event fires, so save it off in a private member variable

public partial class Page : UserControl { private TimelineMarkerRoutedEventArgs tmreCache; void myPlayer_MarkerReached( object sender, TimelineMarkerRoutedEventArgs e ) { //… tmreCache = e;

Finally, the MarkerReached event handler makes the ShowMore button visible and enabled. Here's the complete event handler:

void myPlayer_MarkerReached( object sender, TimelineMarkerRoutedEventArgs e ) { Timer t = new Timer( EndShowMore, // call back ShowMore, // state 2000, // dueTime System.Threading.Timeout.Infinite // period ); tmreCache = e; ShowMore.Visibility = Visibility.Visible; ShowMore.IsEnabled = true; }

This won't compile if you don't at least stub out the call back, which must be a static method,

private static void EndShowMore( object state ) { }

Show More Information

Before we implement the handler for the call back when the ShowMore button will be hidden again, let's explore what might happen if the user clicks on the button while it is visible.

The plan is to have the video player pause, to have the Media player become visible and to have it start showing the appropriate video clip based on which marker caused the button to appear.

We've made the button visible in MarkerReached, we need an event handler for clicking the button, which we set up back in the constructor,

ShowMore.Click += new RoutedEventHandler( ShowMore_Click );

We can start by stubbing out the logic of this event handler,

void ShowMore_Click( object sender, RoutedEventArgs e ) { // pause the player // make the MediaElement visible // determine which marker was reached and set the source for // the media player accordingly // Tell the MeidaElement to play the clip }


We can now add code for each step

void ShowMore_Click( object sender, RoutedEventArgs e ) { // pause the player myPlayer.Pause(); // make the MediaElement visible VideoLink.Visibility = Visibility.Visible; // determine which marker was reached and set the source for // the media player accordingly string prefix = @"C:/Demo/HyperVideoPresentation2/Clips/"; switch ( tmreCache.Marker.Text.ToLower().Trim() ) { case "bindingengine": VideoLink.Source = new Uri( prefix + "bindingengine.wmv" ); break; case "bindingSource": VideoLink.Source = new Uri( prefix + "bindingSource.wmv" ); break; case "logo": VideoLink.Source = new Uri( prefix + "logo.wmv" ); break; case "matchingsrctarget": VideoLink.Source = new Uri( prefix + "matchingsrctarget.wmv" ); break; case "target": VideoLink.Source = new Uri( prefix + "target.wmv" ); break; case "title": VideoLink.Source = new Uri( prefix + "title.wmv" ); break; } // Tell the MeidaElement to play the clip VideoLink.Play(); }

We need to handle the possibility that the Media we're trying to play in the MediaElement will fail, or more likely that the user will tire of watching and will click on it, or for that matter that the clip will come to an end.

Using A Common Event Handler

In all of these cases, we want to stop what is playing, hide the MediaElement and resume playing our primary video. Once again, we put a few event handler declarations into the constructor:

VideoLink.MediaEnded += new RoutedEventHandler( VideoLink_ReturnToMainVideo ); VideoLink.MediaFailed += new EventHandler<exceptionroutedeventargs> ( VideoLink_ReturnToMainVideo ); VideoLink.MouseLeftButtonUp += new MouseButtonEventHandler ( VideoLink_ReturnToMainVideo );

Intellisense will try to give each of the three its own event handler, but we need only one common handler. That is not unusual, but we do have a somewhat uncommon situation in that the three event hander types are different, and thus the second argument would normally be different. If you just let Intellisense create the three methods they look like this:

void VideoLink_MouseLeftButtonUp( object sender, MouseButtonEventArgs e ) void VideoLink_MediaFailed( object sender, ExceptionRoutedEventArgs e ) void VideoLink_MediaEnded( object sender, RoutedEventArgs e )

If this is to compile with a shared event handler for all three, the second argument must be a shared base class,

void VideoLink_ReturnToMainVideo( object sender, RoutedEventArgs e )

Both MouseButtonEventArgs and ExceptionRoutedEventArgs inherit from RoutedEventArgs as shown in the documentation reproduced in figure 10-26.

Figure 10-26. Event Args

Your other obvious choice would be to go all the way up to EventArgs, but there is no obvious advantage.

Within this shared event handler you can take care of the logic of shutting down the MediaElement and resuming the main video

void VideoLink_ReturnToMainVideo( object sender, RoutedEventArgs e ) { VideoLink.Stop(); VideoLink.Visibility = Visibility.Collapsed; myPlayer.Play(); }

Handling the Callback

All that is left now is to handle the callback from the timer. Here the problem is that you have a static method that must change the state of a UI object and they are in different threads.

The Dispather object has a method, Invoke, specifically designed to handle solving this problem, and it does so asynchronously. You pass in a delegate and it executes the associated method in the correct thread. We'll use anonymous methods to set the properties in the UI thread,

Remember that when we started the thread, we passed in the Button as the state object. We begin by casting the state object argument back to type Button, and calling the BeginInvoke method on the Dispatcher associated with that button (which, of course, is in the UI thread)

private static void EndShowMore( object state ) { Button btn = (Button) state; btn.Dispatcher.BeginInvoke( delegate() { btn.IsEnabled = false; } ); btn.Dispatcher.BeginInvoke( delegate() { btn.Visibility = Visibility.Collapsed; } ); }

Using a Lambda Expression

Rather than using an explicit anonymous delegate, we can clean this up a bit by using lambda expressions,

Button btn = (Button) state; btn.Dispatcher.BeginInvoke( () => btn.IsEnabled = false ); btn.Dispatcher.BeginInvoke( () => btn.Visibility = Visibility.Collapsed );

Running the Application

To see this application at work, once again you'll copy the Default.html from the safety of the Demo directory into your debug directory and double click on it. When the Show Button appears you can wait and watch it disappear (proving our callback with its Lambda Expression did its work) or you can click on the button and watch the associated video appear,

Figure 10-27. Hyper Video Playing (Click to view full-size image)

In Figure 10-27 you can see that the user has clicked on the Show More button (which is about to disappear), the video in the main player above has paused and the linked video has appeared below.

A Real World Application

Eric Mork (of Sparkling Client) has created a site with videos using Joel Neubeck & Tim Heuer's video player that makes terrific use of markers and a form of hyper-video.

Eric creates short teaching videos, and embeds markers in each. When the video hits each marker, the table of contents is updated in a separate but related window, and the code associated with what Eric is teaching is shown in a third window. He doesn't link off to supplemental information (yet?) but he does use the markers in ways that extend the user's experience of the material being presented.

Figure 10-28. Using Markers to Enhance a Video (Click to view full-size image)

[1] Alfonso Bedoya to Humphrey Bogart in The Treasure of the Sierra Madre, 1948. (adapted) http://tinyurl.com/nobadges

[2] Old joke from when I was a little kid: "Q: How do you sculpt an elephant?" Answer: "Take a big block of marble, and chip away everything that doesn't look like an elephant."