Genvid Forum

SubmitGameData seemingly drops frames sometimes


#1

Hello,

I got another small problem, although I’m not sure what the cause is . SubmitGameData() “randomly” drops frames from time to time.

For testing I logged the last four bytes of the submission-timecode for each Submit into a file, and also sent that in the payload. I submit my VideoData at the exact same timecode.

In JavaScript I then logged every payload I get in client.onStreamsReceived.

The result is here: (1)

Red lines indicate a dropped frame. The first submitted timecode is 6419400.

For reference, the relevant code looks like this: (2)

My suspicion was that I submit at the frame-boundary and sometimes one frame “swallows” the data meant for the next frame. However, that hypothesis didn’t seem to hold up, because even with code like this:

        private void SubmitVideoData()
        {
            Thread.CurrentThread.Priority = ThreadPriority.Highest;
            
            while (true)
            {
                SubmitVideoDataEvent.WaitOne();
                TakeScreenshot();

                if (CurrentScreenshot != null)
                {
                    var tc = GenvidSDK.GetCurrentTimecode();
                    if (this.Submitted > 10)
                    {
                        tc += 10;
                    }

                    GenvidSDK.SubmitVideoData(tc, VideoStream, CurrentScreenshot);
                    GenvidAnnotator.Annotate(tc);
                }
            }
        }

I still got dropped frames: (3)

Is there any way to find out why my frames are dropped?

Oh, and of course:

        public int FPS = 30;
        /// ...
            Check(GenvidSDK.SetParameter(VideoStream, "framerate", FPS));

That my timer is fairly accurate is obvious from the submitted timecodes. While the system is under considerable load (~80%) when encoding, I’m not sure if that’s the problem. And even with a 10FPS-encoding-session there was still one frame that got dropped in the test I ran: (4)

For now I’ll just disable delta-encoding). If this is a major problem, I’ll fix it to encode a value if it differs from any of the last two frames instead of only checking against the last frame.


#2

Hi Moritz,

Genvid guarantees for our data streams is that the last received frame will always be available. This is necessary to support viewer connecting in the middle of a stream, etc. Annotations have a better guarantee (all annotations are send but the viewer only see the ones actually playback), but still not adequate for delta-encoding.

Right now, the decision about which frames are dropped or not is a bit too much arbitrary, and we are currently developing a more deterministic approach. However, dropping can happen at a later stage and so I don’t expect that delta encoding will ever be adequate on the user side (we will maybe use it internally, but not right now).

Actually, the best way to avoid repeating too much data everytime is to create multiple data streams, with different framerate. This way, the internal protocol can detect if a frame have not change and only repeat it if necessary. Note that such optimization is not actually implement right now however, but will likely be in the future.


#3

I’ll disable delta-encoding for now. It would be trivial to change it to use the last two frames. Full-Updates for onboarding new viewers were sent every second regardless.

We’ll see if there will be any problems if we disable delta-encoding, and if there are I already have a few ideas that could be tried out and benchmarked with your information in mind.

Thank you!


#4

Note that we already resend the last data if it get older than 2 seconds. Future versions will probably be more intelligent (only sending it on new connection), but right now it was enough for our needs.