Adaptive Mic Recording (Fill glap with silence to keep accurate time).

Feb 7 at 2:07 AM
Okay let's see if i can explain this.


As you may know recording an audio source isn't perfect, at least not in C#.
And by that i mean, it's not flawless, and that's to be expected, as it's not made to be that in the way it works.


By flawless i mean if anything interrupt the mic recording thread, or rather takes priority of it for some reason at any case, it will be left unfeeded, and the result will be that the audio supposed to be there isn't and it will just continue.

This can happen more occasionally with a low buffer.

Normally the solution would be to increase the buffer, but i can't do that as i am using it for Network Chat.


Now here is the thing, in the data sent through the network i don't care if audio get's skipped sometimes or not, as the timing doesn't really matter it will be rather accurate anyway in the end.


But for recording this makes all the difference.

If i record 100 seconds, and i lose 10sec, it will become 90, and there is no way to know where it was lost as it can be anywhere and by small amounts.


So, what i want to do is make an adaptive way to record, meaning if a buffer is lost, which i think will always be the amount of the buffer (10ms = 10ms lost), i want to make it write 10ms silence.


So even though the audio may be lost, the time and flow will be accurate.


Hope i explained it well enough, thanks!
Feb 7 at 3:46 PM
If you want to fill in the empty samples you should use the BufferedWaveProvider
/// <summary>
/// Provides a buffered store of samples
/// Read method will return queued samples or fill buffer with zeroes
/// Now backed by a circular buffer
/// </summary>
Feb 9 at 5:58 AM
How can i use that to Save to a .Wav file?

I am using it to playback the audio that's received by just feeding the buffer and playing it.

But how can i save the my recording into a buffer and than save the buffer instead of the actual recording feed?
Feb 9 at 8:27 AM
Use the BufferedWaveProvider with a WaveFileWriter to save the samples to a WAV file.

Have a look at the NAudio Samples to see how it works.
Feb 11 at 11:25 PM
Can you show an example?
I know how to add the mic data to a BufferedWaveProvider, but not how to write what's in the Provider with WaveFileWriter.

Couldn't find any Samples either.
Feb 12 at 1:36 AM
Edited Feb 12 at 1:41 AM
            BufferedWaveProvider b = new BufferedWaveProvider(waveFormat);
            WaveFileWriter.CreateWaveFile(@"C:\buff.wav", b);
Search the nAudio source for BufferedWaveProvider and CreateWaveFile for more examples of the usage.
Feb 12 at 2:30 AM
Doesn't that only works if everything is in the buffer at once?
I need to append as time goes,

waveWriter.Write(data, 0, data.Length);

Like that, just adding data all the time.
Feb 12 at 2:48 AM
I'm not sure if CreateWaveFile waits for the data, if it doesn't try it using .Write instead.
Feb 12 at 3:42 AM
Problem with Write is i can't choose the buffer, i need to choose sample/samples or byte[], and i don't know how to tell it to use the buffer data.
Feb 12 at 4:26 AM
BufferedWaveProvider has a Read() function so send those bytes to the Wavewriter write function.
Feb 12 at 4:37 AM
How, the Read function will only return the count bytes, a number not any data?
Feb 12 at 4:39 AM
You would need to read the data to your own byte[] array first, then write that array to WaveWriter
Feb 12 at 4:44 AM
Doesn't work as intended, anything wrong?

                        RecordingProvider.AddSamples(e.Buffer, 0, e.BytesRecorded);
                        waveWriterYour = new WaveFileWriter(path + Environment.UserName + " - " + DateTime.Now.ToString("yyyy-MM-dd HH-mm-ss-fff") + ".wav", new WaveFormat(48000, 16, 2));

                            byte[] temp;

                            temp = new byte[RecordingProvider.BufferedBytes];
                            RecordingProvider.Read(temp, 0, temp.Length);
                            waveWriterYour.Write(temp, 0, temp.Length);
Feb 12 at 5:45 AM
Looks ok to me but I can't confirm that is the correct way of doing it, perhaps Mark can assist further.
Feb 12 at 6:45 AM
Feb 12 at 7:50 AM
Well those are what i usually do, but the problem is, if the buffer skipped, it will skip writing, making the timing inaccurate.
Coordinator
Feb 12 at 7:56 AM
here's how I typically handle this situation. Timestamp each buffer as it arrives, as accurately as possible (depending on the app, the timestamp could even be sent as part of the audio packet). The receiving end should compare the timestamp received with the timestamp it was expecting (i.e. the last timestamp plus the duration of audio in the last buffer). If the expected time is close enough to the received time (e.g less than 10ms), then simply write directly to the file. If however, the received time is greater than the received time by more than a certain amount, then pad with some silence.
Feb 12 at 8:37 AM
Edited Feb 12 at 8:38 AM
That would be ideal, but i don't know how to do that.

I can see it in theory as you explain it, but not sure how to properly make use of it.

If i simply write all the Mic recorded data to a file, how can i know the time of it?
I am pretty sure every feed should be the same as the buffer, meaning it should always be 10ms for example, and more or less means something is wrong, right?
Feb 12 at 11:04 AM
Edited Feb 12 at 11:46 AM
Well you could use a Timer to observe the incoming data every 10ms or so, if there is no data available, insert 10ms of blank audio. If there is valid data, write it to the writer. If there is more than 10ms of data, adjust the time stamp accordingly.

I was under the impression BufferedWaveProvider did this automatically (i.e fill in empty samples when there is no data) but I guess that is not the case or not suitable for this situation.
Feb 13 at 5:47 AM
That's what i am trying to do know, not really sure how to do it.
                            if (Record)
                            {
                                if (FileCreated != 0)
                                {
                                    if (calltimer.IsRunning == false)
                                        CallTimeThread.Start();
                                    if (TimeSpan.FromMilliseconds(TestTimer / 192).Milliseconds < calltimer.ElapsedMilliseconds - 5000)
                                    {
                                        TestTimer = calltimer.ElapsedMilliseconds;
                                        int silence = (int)calltimer.ElapsedMilliseconds - TimeSpan.FromMilliseconds(TestTimer / 192).Milliseconds;
                                        TestTimer = calltimer.ElapsedMilliseconds * 192;
                                        int avgBytesPerMillisecond = 192;
                                        var silenceArraySize = avgBytesPerMillisecond * silence;
                                        byte[] silenceArray = new byte[silenceArraySize];
                                        waveWriterYour.Write(silenceArray, 0, silenceArray.Length);
                                    }
                                    this.BeginInvoke((Action)(() =>
                      {
                          RecTime.Text = TimeSpan.FromMilliseconds(TestTimer / 192).ToString();
                      }));
                                    waveWriterYour.Write(AudioData2, 0, AudioData2.Length);
                                    waveWriterYour.Flush();
                                }

                            }
This completely fails, but i guess it's somewhere on the line.

The TestTimer will be exactly the same time as the audio, meaning the timer itself is correct.
The other, CallTimer will begin at about the same time and just measure time itself, so when they are off, it means audio has been skipped (Though they aren't precise when they start, but at least it's pretty close, and can probably be done better.


Now i just want it to say, "If TestTimer is 100ms less than Calltimer, add 100ms Silence" , or something like that.

Which should make it fairly accurate, at least in theory.
Feb 13 at 8:47 AM
Try using .TotalMilliseconds instead of .Milliseconds

Any length over 1 second would be lost.
Feb 13 at 8:02 PM
It seems to work:
            if (Record && connect)
                TestTimer += e.Buffer.Length / 192;
            else
                TestTimer = 0;
            if (!RecordTimer.IsRunning && Record)
            {
                RecordTimer.Reset();
                RecordTimer.Start();
            }
            else
                RecordTimer.Stop();

            if (TimeSpan.FromMilliseconds(TestTimer).TotalMilliseconds < calltimer.ElapsedMilliseconds)
            {

                int silence = (int)calltimer.ElapsedMilliseconds - (int)TimeSpan.FromMilliseconds(TestTimer).TotalMilliseconds;
                TestTimer += silence;
                int avgBytesPerMillisecond = 192;
                var silenceArraySize = avgBytesPerMillisecond * silence;
                byte[] silenceArray = new byte[silenceArraySize];
                SendQueue.Add((byte[])silenceArray);
            }

Where SendQueue will have the Audio added as well later, and then produces and sends the Audio.

It seems to work when i try to lose much audio data, and it still seems to be synced, probably not perfect, but i think it does the trick.

Anything that may be wrong with it, or do you think it will be alright?

PS: Yeah 1 second will be lost if it's 1000ms as it goes back to 0 again, but i don't think 1 second can ever be lost, except if something extreme happens, like a total PC Freeze, i think 100-200ms is about the most, and even that is drastic, probably 10-30ms is more realistic.
Feb 14 at 9:20 PM
Edited Feb 14 at 9:41 PM
Hmm it doesn't seem to work as expected, it seems that a chat will go out of sync when i record it, at least when there is a lot going in, which i guess causes skipped buffers.

The network should be flawless when it comes to the Data as it's TCP, meaning it must be somewhere on the Recording side.

It only seem to occurs as before when the PC is very active, and non existent if nothing is going on, which means my "fail safe" doesn't seem to do it's part.

The case seems to be that my microphone Recording is shorter (Even though the length is the same), then the received recording, though that can probably change depending on which pc skips the most at the time.

My Silence adding seems to work, which makes me confused at the moment.
Feb 17 at 12:51 PM
Not sure but I suggest you go back a step and test the basic functionality of each main function. For example make sure the input data is arriving as expected, ensure SendQueue is working as expected, add a lot of debug checks, etc..
Feb 17 at 3:13 PM
I may have solved it, i think i made something wrong with the calculations, however here is my other version:
 private void Sending(object sender, NAudio.Wave.WaveInEventArgs e)
        {
            try
            {
                TestTimer += e.Buffer.Length / 192;

                if (!RecordTimer.IsRunning)
                {
                    RecordTimer.Reset();
                    RecordTimer.Start();
                }
                skip = false;
                int silence = (int)(RecordTimer.ElapsedMilliseconds - (TestTimer));
                if (silence > 10 || silence < -10 && RecordTimer.IsRunning)
                {


                    if (silence > 0)
                    {
                        TestTimer += silence;
                        int avgBytesPerMillisecond = 192;
                        var silenceArraySize = avgBytesPerMillisecond * silence;
                        byte[] silenceArray = new byte[silenceArraySize];
                        SendQueue.Add((byte[])silenceArray);
                    }
                    else
                    {
                        TestTimer -= e.Buffer.Length / 192;
                        skip = true;
                    }

                }

                if (connect && MuteMic.Checked == false && skip == false)
                {
                    SendQueue.Add((byte[])e.Buffer.Clone());
                }
                if (Record)
                {
                    if (FileCreated == 0)
                    {
                        waveWriterYour = new WaveFileWriter(path + Environment.UserName + " - " + DateTime.Now.ToString("yyyy-MM-dd HH-mm-ss-fff") + ".wav", new WaveFormat(48000, 16, 2));
                        FileCreated = 1;
                    }

                }

            }
            catch (Exception ex)
            {
                MessageBox.Show(ex.Message, ":Sending", MessageBoxButtons.OK, MessageBoxIcon.Error);
            }
        }
This should force the Audio to follow the Stopwatch within a 30ms area, and by that i think it should be synced withing that as well.
Hopefully this will work as expected.
Feb 19 at 8:03 AM
It seems to work when i did some tests.

Though i am still at a wonder why BufferedWaveProvider is always increasing in buffer making me force clearing it.

Could someone please try to feed your mic (or any other recording device) to a BufferedWaveProvider and check the buffer, does it stay the same all the time, or does it eventually increase?

Cause in theory, it should always be the same, as the data is going to be played the same speed it gets received.
Feb 19 at 9:47 AM
That is what BufferedWaveProvider does, fill in zeroes when there is no sample queue. Perhaps you need to use a different WaveProvider or wait for Mark to advise you further.
Feb 19 at 9:58 AM
It doesn't seem to work as intended, as i see no meaning in adding way to much silence, it goes far beyond what's actually missed.
Even if i try to skip data from the recording to sync to the BufferedWaveProvider it won't change, it just adds the buffer to it's own accord, i have no way to control it.

I would gladly like to try another one.

The best thing would be like a MemoryStream, you keep adding the data, and keeps playing it, and then removes what has been played.
This is however how BufferedWaveProvider works i think, except for the part where it adds zereos all over the place.
Feb 19 at 10:16 AM
I'd try WaveProvider16, if that doesn't work as expected you could always create your own WaveProvider that suites your needs, perhaps using a MemoryStream as you mentioned.
Feb 19 at 9:24 PM
How am i supposed to use WaveProvider16, i can't add samples or anything to it, so i am a bit confused?
Feb 19 at 11:02 PM
You would need to modify it to suite your needs, add your own AddSamples() function and a MemoryStream or a List collection of samples that will be read by the output interface (wavefilewriter).
Coordinator
Feb 19 at 11:17 PM
I think you're overcomplicating this. No need for an extra thread. When you get a DataAvailableEvent, write immediately to the file while still in the handler. Here;s some (completely untested and very rough) pseudo code to give you an idea. You could use Stopwatch instead of DateTime for more accuracy.
void OnDataAvailable(object sender, WaveInEventArgs e) 
{
    var timeStamp = DateTime.NowUtc;
    // insert silence if needed (todo: don't insert silence on first buffer)
    if (timeStamp > expectedTime.AddMilliseconds(20)) // some
    {
        var millisecondsToInsert = (timeStamp - expectedTime),TotalMilliseconds;
        var silence = new byte[bytesPerMillisecond * millisecondsToInsert];
        writer.Write(silence, 0, silence.Length);
    }
    // now write the audio
    writer.Write(e.Buffer, 0, e.BytesRecorded);
    // now set up the expected time of the next buffer
    expectedTime = timeStamp.AddMilliseconds(e.BytesRecorded / bytesPerMillisecond);
}
Feb 20 at 1:56 AM
Why i am using an Extra Thread is simply to make it easier as i send the Mic data, and i write it, so if i just add the data to a Queue (Silence or real data) it will be in order on that thread anyway.

It also takes of some pressure on the DataAvailable thread, not sure if that matters though.


I prefer using a StopWatch as you said yourself for accuracy, but here is my attempt to your code, but with a StopWatch, it doesn't work as intended however.
        private void Sending(object sender, NAudio.Wave.WaveInEventArgs e)
        {
            try
            {
                var bytesPerMillisecond = 192;
                if (!RecordTimer.IsRunning)
                {
                    TestTimer = 0;
                    RecordTimer.Reset();
                    RecordTimer.Start();
                    if (RecordTimer.ElapsedMilliseconds < 9)
                        Thread.Sleep(1);
                }
                TestTimer += e.Buffer.Length / 192;

                var timeStamp = RecordTimer.Elapsed;
                this.BeginInvoke((Action)(() => { SilenceText.Text = String.Format("Silence: {0} ms", (timeStamp - expectedtime).TotalMilliseconds); }));

                if (timeStamp.Milliseconds > (expectedtime.Milliseconds + 20))
                {
                    double millisecondsToInsert = (timeStamp - expectedtime).TotalMilliseconds;
                    byte[] silence = new byte[bytesPerMillisecond * (int)millisecondsToInsert];
                    SendQueue.Add(silence);
                }

                if (connect && MuteMic.Checked == false && skip == false)
                {
                    SendQueue.Add((byte[])e.Buffer.Clone());
                }
                if (Record && FileCreated == 0)
                {
                    waveWriterYour = new WaveFileWriter(path + Environment.UserName + " - " + DateTime.Now.ToString("yyyy-MM-dd HH-mm-ss-fff") + ".wav", new WaveFormat(48000, 16, 2));
                    FileCreated = 1;
                }
                expectedtime = timeStamp.Add(TimeSpan.FromMilliseconds(e.BytesRecorded / bytesPerMillisecond));
            }
            catch (Exception ex)
            {
                MessageBox.Show(ex.Message, ":Sending", MessageBoxButtons.OK, MessageBoxIcon.Error);
            }
        }
Feb 20 at 12:38 PM
I have been looking at the Source Code on BufferedWaveProvider, but i can't see what part adds Silence, it just looks like a normal Add/Read/Clear class, nothing fancy automatic operation?
Coordinator
Feb 20 at 12:56 PM
BufferedWaveProvider doesn't add silence, it just returns silence when you read if there's nothing in the buffer. So if you have something reading in real-time from it, then it will handle the silence adding automatically.
Feb 20 at 1:02 PM
Well in the end it will be the same though, silence will get "added", well that itself isn't a problem, however, the problem is, the BufferedWaveProvider will have A lot more "audio time" compared to the real audio time, it can be seconds (which i think is much) in the end.

My "solution" is to Clear the buffer every time it get´s to 120ms, or similar, and that isn't actually a good solution.


From what you say, this shouldn't be the case if it's only playing silence when nothing is there, and when something is added it will be playing from that, which in should be something like:

100ms data added,
100ms played,
nothing added for 500ms,
Provider returns 500ms of silence (one buffer of silence at a time i guess),
100ms data added,
skip the silence buffer currently active and start playing the data.


And if that´s how it works, i can't see how the buffer can always be bigger than the actual data.
Feb 22 at 8:49 AM
Mark, is there a way to save the BufferedWaveProvider's buffer?

I would like to analyze it and compare it to a direct copy of the stream itself.
Coordinator
Feb 24 at 8:13 AM
you can either write to a WaveFileWriter at the same time as putting data into the BufferedWaveProvider, or, if you want to save any inserted silence, then I usually make a very simple pass-through IWaveProvider that in its Read method reads from its source, writes to a WaveFileWriter, and then passes the audio through.
Feb 24 at 11:26 PM
I know a way to do that, read the Buffer and put that in a byte[], however, i can't make it do it completely to always work.

Meaning, if it reads it again, it can read duplicated data (if the buffer hasn't been playbacked yet).

So do you have a way to make the whole operation automated?

I already get the data from TCP, and i write that data untouched.
I only need to write another one wiht the WaveProviders data, and compare them.
Coordinator
Feb 27 at 10:14 PM
it really is very simple. In your derived waveprovider's read method just do this
public int Read(byte[] buffer, int offset, int count) 
{
    int bytesRead = sourceProvider.Read(buffer, offset, count);
    waveFileWriter.Write(buffer, offset, bytesRead);
    return bytesRead;
}
Feb 28 at 4:24 AM
Edited Feb 28 at 4:52 AM
Should i do that Before i add samples or after (i am guessing after).

And if so, it does seems like reading will clear the buffer it has read, so i am guessing i musn't play from the waveprovider at the same time?

EDIT:


Got it to work, but weird enough, it's pretty much identical to the source.

Something must happen that's difference when Reading and Playing from the buffer.

I guess the "Automatic Silence" comes when Playing, and Reading just Reads the Data as told.

Would it be possible to Read the data as it's playing, as i would like to get that "Automatic Silence".
Coordinator
Feb 28 at 7:54 AM
You just put this in your signal chain e.g.: BufferedWaveProvider -> CustomWavWritingProvider -> WaveOutEvent

The output device is doing all the calls to Read, and BufferedWaveProvider is doing the automatic silence.
Feb 28 at 10:53 AM
But how can i write, and then tell it to play it?

Cause currently, if i add samples to the buffer and then write it, i will not hear anything, which i guess means reading clears it.
Coordinator
Mar 4 at 7:17 PM
The wave player is the only thing calling Read. The custom wav writing provider I showed above happens to write it to the WAV as it is being pulled through
Mar 6 at 4:18 PM
Edited Mar 6 at 4:22 PM
The way i do recording:
i keep wave in recording always on, but gathering of mic data is only on user command
i send microphone wave in to buffered wave provider,
then send buffered wave provider to my custom dsp_after_mic Effect class (which is as ISampleProvider),
which actually read byte[] 16 bit mic data from buffered, converts it , and outputs float[] buffer
then dsp_after_mic is sent to mixer (MixingSampleProvider)
(if dsp_after_mic read less 16bit samples than mixer requested, remaining floats for mixer float buffer are filled with 0.0f) so it always return number of floats requested
mixer is then sent to dsp_after_mixer, and then to waveout
but dsp_after_mixer also make fork copy of buffer and write it to another buffered wave provider called buffered_dsp_outstream_fork that is later read by encoder and by tcp if needed.

For mic buffered discard on overflow is true, also true for outsream_fork.
On start of gathering mic data, i call Clear of mic buffered wave provider and add dsp_after_mic to mixer.
On stop of gathering mic data, i remove dsp_after_mic from mixer.
Mar 15 at 12:54 AM
That sadly, is over my capability, i can only hardly grasp what you mean;S


However, i noticed a thing.

This may be worth looking into Mark.


If i simply let ASIO/Wasapi play from a Wavein, and that wave in is an Waveinevent (or well doesn't matter i guess, can be run on GUI as well probably).

Then it should ALWAYS be delayed the same, as no factor involved should make it increase the delay or anything like that, correct?

Though even so, the audio will be delayed more over time, not sure if it has a limit or not, but it will differ from start, to what it can reach.
Not sure how much ms we are talking about, but it's noticeable, but probably not over 100ms, but it may depend.


Now this is Without any BufferedWaveProvider involved, and only pure, input/output, meaning everything should be untouched and clean.


Do you have any idea why this is the case Mark, cause the only thing i can think of, is that the Data is touched by something, or that the flow isn't correct.
My guess is that Recording may be done faster then Playing which causes a "buffer?" to build up. And this shouldn't happen.