Roy' Tech

카테고리 없음

ExoPlayer 동작 분석 (prepare)

Roien 2021. 12. 19.
반응형

Simple example code

class MainActivity : AppCompatActivity() {
    companion object {
        const val TAG = "MainActivity"
    }

    inner class PlayerStateListener : Player.EventListener {
        override fun onPlaybackStateChanged(state: Int) {
            when (state) {
                ExoPlayer.STATE_IDLE -> Log.d(TAG, "STATE_IDLE")
                ExoPlayer.STATE_BUFFERING -> Log.d(TAG, "STATE_BUFFERING")
                ExoPlayer.STATE_READY -> Log.d(TAG, "STATE_READY")
                ExoPlayer.STATE_ENDED -> Log.d(TAG, "STATE_ENDED")
            }
        }
    }

private lateinit var playerView: PlayerView
private var player: ExoPlayer? = null
private val listener = PlayerStateListener()

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        playerView = findViewById(R.id.playerView)
}

    fun onClickPlay(view: View) {
        if (player == null) {
            player = SimpleExoPlayer.Builder(this).build()
            playerView.player = player
        }

        player?.apply {
            if (isPlaying) {
                stop()
            }

            setMediaItem(MediaItem.fromUri("asset:///video/20211028_134942113.mp4"))
            //addMediaItem(MediaItem.fromUri("asset:///video/20211028_134942113.mp4"))
            addListener(listener)
            prepare()
            play()
            seekTo(2000L)
        }
    }

    fun onClickRelease(view: View) {
        player?.apply {
            stop()
            release()
            removeListener(listener)
        }

        player = null
    }

기본 player 생성 (별도 설정 없이) 후, MediaItem을 지정한다. 

이후 prepare -> play를 수행한다. 

멈춰야할 경우 stop -> release를 수행한다. 

 

1. Source 생성

1) Media source의 종류

media source types

Ÿ   DashMediaSource

Ÿ   HlsMediaSource

Ÿ   RtspMediaSource

Ÿ   SsMediaSource

Ÿ   ProgressiveMediaSource

Ÿ   FakeMediaSource

Ÿ   AdsMediaSource

Ÿ   SilenceMediaSource

Ÿ   MergingMediaSource

Ÿ   SingleSampleMediaSource

Ÿ   MaskingMediaSource

Ÿ   ConcatenatingMediaSource

Ÿ   LoopingMediaSource

Ÿ   ClippingMediaSource

 

2) HLS Media Source 생성

private static HlsMediaSource.Factory createHlsMediaSourceFactory(
      String playlistUri, String playlist) {
    FakeDataSet fakeDataSet = 
new FakeDataSet().setData(playlistUri, Util.getUtf8Bytes(playlist));
    return new HlsMediaSource.Factory(
            dataType -> new FakeDataSource.Factory().setFakeDataSet(fakeDataSet).createDataSource())
        .setElapsedRealTimeOffsetMs(0);
}

String playlistUri = "fake://foo.bar/media0/playlist.m3u8";
String playlist =
        "#EXTM3U\n"
            + "#EXT-X-PROGRAM-DATE-TIME:2020-01-01T00:00:00.0+00:00\n"
            + "#EXT-X-TARGETDURATION:4\n"
            + "#EXT-X-VERSION:3\n"
            + "#EXT-X-MEDIA-SEQUENCE:0\n"
            + "#EXTINF:4.00000,\n"
            + "fileSequence0.ts\n"
            + "#EXTINF:4.00000,\n"
            + "fileSequence1.ts\n"
            + "#EXTINF:4.00000,\n"
            + "fileSequence2.ts\n"
            + "#EXTINF:4.00000,\n"
            + "fileSequence3.ts\n"
            + "#EXT-X-SERVER-CONTROL:CAN-SKIP-UNTIL=24";

SystemClock.setCurrentTimeMillis(Util.parseXsDateTime("2020-01-01T00:00:17.0+00:00"));
HlsMediaSource.Factory factory = createHlsMediaSourceFactory(playlistUri, playlist);
MediaItem mediaItem = MediaItem.fromUri(playlistUri);
HlsMediaSource mediaSource = factory.createMediaSource(mediaItem);

Timeline timeline = prepareAndWaitForTimeline(mediaSource);

Timeline.Window window = timeline.getWindow(0, new Timeline.Window());

HlsMediaSource.Factory를 사용해서 HlsMediaSourceFactory를 생성한다. 

HlsMediaSource 생성 후 이를 Timeline을 생성하고, 

Timeline으로 Winodw를 생성한다.

 

이렇게 생성한 Media Source를 Player에 set 한다.

SimpleExoPlayer player = new SimpleExoPlayer.Builder(getApplicationContext()).build();

player.setRepeatMode(Player.REPEAT_MODE_ALL);
player.setMediaSource(mediaSource);
player.prepare();
player.play();

VideoProcessingGLSurfaceView videoProcessingGLSurfaceView =
        Assertions.checkNotNull(this.videoProcessingGLSurfaceView);
videoProcessingGLSurfaceView.setVideoComponent(
        Assertions.checkNotNull(player.getVideoComponent()));

 

HlsMediaSourceFactory는 DefaultDrmSessionManager, DefaultHlsPlaylist를 사용한다.

public static final class Factory implements MediaSourceFactory {
public Factory(HlsDataSourceFactory hlsDataSourceFactory) {
      this.hlsDataSourceFactory = checkNotNull(hlsDataSourceFactory);
      drmSessionManagerProvider = new DefaultDrmSessionManagerProvider();
      playlistParserFactory = new DefaultHlsPlaylistParserFactory();
      playlistTrackerFactory = DefaultHlsPlaylistTracker.FACTORY;
      extractorFactory = HlsExtractorFactory.DEFAULT;
      loadErrorHandlingPolicy = new DefaultLoadErrorHandlingPolicy();
      compositeSequenceableLoaderFactory =
   new DefaultCompositeSequenceableLoaderFactory();
      metadataType = METADATA_TYPE_ID3;
      streamKeys = Collections.emptyList();
      elapsedRealTimeOffsetMs = C.TIME_UNSET;
    }

 

 

public MediaSource createMediaSource(MediaItem mediaItem) {

@C.ContentType
    int type =
        Util.inferContentTypeForUriAndMimeType(
            mediaItem.playbackProperties.uri, mediaItem.playbackProperties.mimeType);

    @Nullable MediaSourceFactory mediaSourceFactory = mediaSourceFactories.get(type);

    설정된 live min/max speed, min/max offset으로 media source를 다시 build

    MediaSource mediaSource = mediaSourceFactory.createMediaSource(mediaItem);

    subtitle이 존재하면, subtitle media source를 만든 후, 
    두 media source를 merge

    return maybeWrapWithAdsMediaSource(mediaItem, maybeClipMediaSource(mediaItem, mediaSource));

 

3) DASH or PD Media Source 생성

@C.ContentType int type = 
Util.inferContentType(uri, intent.getStringExtra(EXTENSION_EXTRA));
if (type == C.TYPE_DASH) {
    mediaSource = new DashMediaSource.Factory(dataSourceFactory)
        .setDrmSessionManager(drmSessionManager)
        .createMediaSource(MediaItem.fromUri(uri));
} else if (type == C.TYPE_OTHER) {
      mediaSource =
          new ProgressiveMediaSource.Factory(dataSourceFactory)
              .setDrmSessionManager(drmSessionManager)
              .createMediaSource(MediaItem.fromUri(uri));
}

 

 

2. prepare

SimpleExoPlayer.prepare

public void prepare() {
    verifyApplicationThread();
    boolean playWhenReady = getPlayWhenReady();
    @AudioFocusManager.PlayerCommand
    int playerCommand = audioFocusManager.updateAudioFocus(playWhenReady, Player.STATE_BUFFERING);
    updatePlayWhenReady(   player.setPlayWhenReady(playWhenReady...
        playWhenReady, playerCommand, getPlayWhenReadyChangeReason(playWhenReady, playerCommand));
    player.prepare();
  }

ExoPlayerImplInternal.prepare

private void prepareInternal() {
    playbackInfoUpdate.incrementPendingOperationAcks(/* operationAcks= */ 1);
    resetInternal(
        /* resetRenderers= */ false,
        /* resetPosition= */ false,
        /* releaseMediaSourceList= */ false,
        /* resetError= */ true);
    loadControl.onPrepared();
    setState(playbackInfo.timeline.isEmpty() ? Player.STATE_ENDED : Player.STATE_BUFFERING);
    mediaSourceList.prepare(bandwidthMeter.getTransferListener());
    handler.sendEmptyMessage(MSG_DO_SOME_WORK);
  }

prepare를 수행하면 (예를들어 HLS playback인 경우) 아래와 같이 source들을 생성한다.

이후 doSomeWork를 주기적으로 수행하면서 renderer가 render를 수행하게 된다.

render가 media source가 생성한 ES stream을 소비하는 역할을 수행한다. 

여기서 소비란 decode 및 출력 rendering을 의미한다.

private void doSomeWork() throws ExoPlaybackException, IOException {
      updatePeriods();
      updatePlaybackPositions();

      if (playingPeriodHolder.prepared) {
          playingPeriodHolder.mediaPeriod.discardBuffer(
             playbackInfo.positionUs - backBufferDurationUs, retainBackBufferFromKeyframe);

          for (int i = 0; i < renderers.length; i++) {
            Renderer renderer = renderers[i];
            renderer.render(rendererPositionUs, rendererPositionElapsedRealtimeUs);

     재생의 finish, buffering, ready 등의 처리 수행

doSomeWork의 동작설명

n  updatePeriods
n  updatePlaybackPositions
n  if playingPeriodHolder.prepared
u      여러 renderer들을 render(rendererPositionUs, …)
n  rendering 종료 중 && 마지막 period 에서
u      setPlayWhenReadyInternal
n  rendering 종료 중 && period holder가 종료
u      stopRenderers
n  buffering 상태
u      setState(Player.STATE_READY);
u      ready 시 재생해야 한다면,
u      startRenderers();
n  ready 상태
u      setState(Player.STATE_BUFFERING);
u      stopRenderers();
n  buffering 이고 READY 상태
u      maybeScheduleWakeup
n  else 아직 종료 아님
u      scheduleNextWork

 

3. MediaCodecRenderer 동작

renderer의 render는

    1) drainOutputBuffer를 호출하여 decode 된 buffer가 있다면 이를 sink로 출력한다.

    2) feedInputBuffer를 수행하여 ES stream을 읽어온다. 

 

읽은 ES stream은 MediaCodec에 전달하여 decode를 수행한다. (queueInputBuffer)

MediaCodec은 ACodec이 관리하는 IOMX component에 decode 수행을 요청한다. 

(fillThisBuffer, emptyThisBuffer와 같은 OMX 표준 API를 사용한다)

 

code flow는 다음과 같다.

ExoPlayerImplInternal.doSomeWork
    renderer.render(
    
MediaCodecRenderer.render
    drainOutputBuffer
    feedInputBuffer

MediaCodecRenderer.feedInputBuffer
    inputIndex = codec.dequeueInputBufferIndex();
    buffer.data = codec.getInputBuffer(inputIndex);

    readSource(formatHolder, buffer, ...)  
        BaseRenderer.readSource
            stream.readData(formatHolder, buffer, ...)

    if buffer.isEncrypted()
        buffer.cryptoInfo.increaseClearDataFirstSubSampleBy(...)

    if hasSupplementalData
        handleInputBufferSupplementalData(buffer);

    onQueueInputBuffer(buffer);
        = MediaCodecAudioRenderer.onQueueInputBuffer
              currentPositionUs = buffer.timeUs;

        = MediaCodecVideoRenderer.onQueueInputBuffer
              onProcessedTunneledBuffer(buffer.timeUs);

    if (bufferEncrypted) 
        codec.queueSecureInputBuffer(inputIdx, 0, buffer.cryptoInfo, ...)
    else
        codec.queueInputBuffer(inputIdx, 0, buffer.data.limt(), ..


MediaCodecRenderer.drainOutputBuffer
    outputIndex = codec.dequeueOutputBufferIndex(outputBufferInfo);
    if eos
        processEndOfStream();

    outputBuffer = codec.getOutputBuffer(outputIndex);
    processOutputBuffer(..., outputBuffer, ....)

    if (processedOutputBuffer)
        onProcessedOutputBuffer(...)
        resetOutputBuffer()
        if eos
            processEndOfStream()

MediaCodecVideoRenderer
    processOutputBuffer
        renderOutputBuffer
        renderOutputBufferV21
    

MediaCodecAudioRenderer
    processOutputBuffer
        AudioSink.handleBuffer -> DefaultAuidoSink.handleBuffer -> AudioTrack.write

 

 

반응형

댓글

티스토리툴바