学步园

十二 h264 rtp包的时间戳

这次我们一起来分析一下live555中是怎样为rtp包打时间戳的．就以h264为例吧．

void H264VideoRTPSink::doSpecialFrameHandling(unsigned /*fragmentationOffset*/,<br /> unsigned char* /*frameStart*/,<br /> unsigned /*numBytesInFrame*/,<br /> struct timeval framePresentationTime,<br /> unsigned /*numRemainingBytes*/)<br /> {<br /> // Set the RTP 'M' (marker) bit iff<br /> // 1/ The most recently delivered fragment was the end of (or the only fragment of) an NAL unit, and<br /> // 2/ This NAL unit was the last NAL unit of an 'access unit' (i.e. video frame).<br /> if (fOurFragmenter != NULL) {<br /> H264VideoStreamFramer* framerSource = (H264VideoStreamFramer*) (fOurFragmenter->inputSource());<br /> // This relies on our fragmenter's source being a "H264VideoStreamFramer".<br /> if (fOurFragmenter->lastFragmentCompletedNALUnit()<br /> && framerSource != NULL && framerSource->pictureEndMarker()) {<br /> setMarkerBit();<br /> framerSource->pictureEndMarker() = False;<br /> }<br /> }</p> <p> setTimestamp(framePresentationTime);<br /> }函数中先检测是否是一个帧的最后一个包，如果是，打上'M'标记．然后就设置时间戳．这个间戳是哪来的呢？需看函数doSpecialFrameHandling()是被谁调用的，经查找，是被MultiFramedRTPSink::afterGettingFrame1()调用的．MultiFramedRTPSink::afterGettingFrame1()的参数presentationTime传给了doSpecialFrameHandling()．MultiFramedRTPSink::afterGettingFrame1()是在调用source的getNextFrame()时传给了source．传给哪个source呢？传给了H264FUAFragmenter,还记得暗渡陈仓那件事吗？所以H264FUAFragmenter在获取一个nal
unit后调用了MultiFramedRTPSink::afterGettingFrame1()．也就是H264FUAFragmenter::afterGettingFrame1()调用了MultiFramedRTPSink::afterGettingFrame1()．
H264FUAFragmenter::afterGettingFrame1()是被它自己的source的afterGettingFrame1()调用的．H264FUAFragmenter的source是谁呢？是H264VideoStreamFramer，是在暗渡陈仓时传给H264FUAFragmenter的构造函数的．
H264VideoStreamFramer的afterGettingFrame1()是没有的，代替之的是MPEGVideoStreamFramer::continueReadProcessin()．它被MPEGVideoStreamParser暗中传给了StreamParser的构造函数．所以StreamParser在分析完一帧（或nal unit)之后，调用的就是MPEGVideoStreamFramer::continueReadProcessin()．以下即是证明：(补充：以下函数并不是在parser分析完一帧（或nal
unit）之后调用，而是parser利用ByteStreamFileSuorce获取到原始数据后调用，然后MPEGVideoStreamFramer再调用Parser的parser()函数分析原始数据)

void StreamParser::afterGettingBytes(void* clientData,<br /> unsigned numBytesRead,<br /> unsigned /*numTruncatedBytes*/,<br /> struct timeval presentationTime,<br /> unsigned /*durationInMicroseconds*/)<br /> {<br /> StreamParser* parser = (StreamParser*) clientData;<br /> if (parser != NULL)<br /> parser->afterGettingBytes1(numBytesRead, presentationTime);<br /> }</p> <p>void StreamParser::afterGettingBytes1(unsigned numBytesRead,<br /> struct timeval presentationTime)<br /> {<br /> // Sanity check: Make sure we didn't get too many bytes for our bank:<br /> if (fTotNumValidBytes + numBytesRead > BANK_SIZE) {<br /> fInputSource->envir()<br /> << "StreamParser::afterGettingBytes() warning: read "<br /> << numBytesRead << " bytes; expected no more than "<br /> << BANK_SIZE - fTotNumValidBytes << "\n";<br /> }</p> <p> fLastSeenPresentationTime = presentationTime;</p> <p> unsigned char* ptr = &curBank()[fTotNumValidBytes];<br /> fTotNumValidBytes += numBytesRead;</p> <p> // Continue our original calling source where it left off:<br /> restoreSavedParserState();<br /> // Sigh... this is a crock; things would have been a lot simpler<br /> // here if we were using threads, with synchronous I/O...<br /> fClientContinueFunc(fClientContinueClientData, ptr, numBytesRead,<br /> presentationTime);<br /> }fClientContinueFunc就是MPEGVideoStreamFramer::continueReadProcessin()，而且我们看到时间戳被传入fClientContinueFunc．
然而，MPEGVideoStreamFramer::continueReadProcessin()中跟本就不理这个时间戳，因为这个时间戳是ByteStreamFileSource计算出来的，它跟本就不可能正确．

void MPEGVideoStreamFramer::continueReadProcessing(void* clientData,<br /> unsigned char* /*ptr*/,<br /> unsigned /*size*/,<br /> struct timeval /*presentationTime*/)<br /> {<br /> MPEGVideoStreamFramer* framer = (MPEGVideoStreamFramer*) clientData;<br /> framer->continueReadProcessing();<br /> }看来真正的时间戳是在MPEGVideoStreamFramer中计算的，但是H264VideoStreamFramer并没有用到MPEGVideoStreamFramer中那些计算时间戳的函数，而是另外计算，其实H264VideoStreamFramer也没有自己去计算，而是利用H264VideoStreamParser计算的．是在哪个函数中呢？在parser()中！

unsigned H264VideoStreamParser::parse()<br /> {<br /> try {<br /> // The stream must start with a 0x00000001:<br /> if (!fHaveSeenFirstStartCode) {<br /> // Skip over any input bytes that precede the first 0x00000001:<br /> u_int32_t first4Bytes;<br /> while ((first4Bytes = test4Bytes()) != 0x00000001) {<br /> get1Byte();<br /> setParseState(); // ensures that we progress over bad data<br /> }<br /> skipBytes(4); // skip this initial code</p> <p> setParseState();<br /> fHaveSeenFirstStartCode = True; // from now on<br /> }</p> <p> if (fOutputStartCodeSize > 0) {<br /> // Include a start code in the output:<br /> save4Bytes(0x00000001);<br /> }</p> <p> // Then save everything up until the next 0x00000001 (4 bytes) or 0x000001 (3 bytes), or we hit EOF.<br /> // Also make note of the first byte, because it contains the "nal_unit_type":<br /> u_int8_t firstByte;<br /> if (haveSeenEOF()) {<br /> // We hit EOF the last time that we tried to parse this data,<br /> // so we know that the remaining unparsed data forms a complete NAL unit:<br /> unsigned remainingDataSize = totNumValidBytes() - curOffset();<br /> if (remainingDataSize == 0)<br /> (void) get1Byte(); // forces another read, which will cause EOF to get handled for real this time<br /> if (remainingDataSize == 0)<br /> return 0;<br /> firstByte = get1Byte();<br /> saveByte(firstByte);</p> <p> while (--remainingDataSize > 0) {<br /> saveByte(get1Byte());<br /> }<br /> } else {<br /> u_int32_t next4Bytes = test4Bytes();<br /> firstByte = next4Bytes >> 24;<br /> while (next4Bytes != 0x00000001<br /> && (next4Bytes & 0xFFFFFF00) != 0x00000100) {<br /> // We save at least some of "next4Bytes".<br /> if ((unsigned) (next4Bytes & 0xFF) > 1) {<br /> // Common case: 0x00000001 or 0x000001 definitely doesn't begin anywhere in "next4Bytes", so we save all of it:<br /> save4Bytes(next4Bytes);<br /> skipBytes(4);<br /> } else {<br /> // Save the first byte, and continue testing the rest:<br /> saveByte(next4Bytes >> 24);<br /> skipBytes(1);<br /> }<br /> next4Bytes = test4Bytes();<br /> }<br /> // Assert: next4Bytes starts with 0x00000001 or 0x000001, and we've saved all previous bytes (forming a complete NAL unit).<br /> // Skip over these remaining bytes, up until the start of the next NAL unit:<br /> if (next4Bytes == 0x00000001) {<br /> skipBytes(4);<br /> } else {<br /> skipBytes(3);<br /> }<br /> }</p> <p> u_int8_t nal_ref_idc = (firstByte & 0x60) >> 5;<br /> u_int8_t nal_unit_type = firstByte & 0x1F;</p> <p> switch (nal_unit_type) {<br /> case 6: { // Supplemental enhancement information (SEI)<br /> analyze_sei_data();<br /> // Later, perhaps adjust "fPresentationTime" if we saw a "pic_timing" SEI payload??? #####<br /> break;<br /> }<br /> case 7: { // Sequence parameter set<br /> // First, save a copy of this NAL unit, in case the downstream object wants to see it:<br /> usingSource()->saveCopyOfSPS(fStartOfFrame + fOutputStartCodeSize,<br /> fTo - fStartOfFrame - fOutputStartCodeSize);</p> <p> // Parse this NAL unit to check whether frame rate information is present:<br /> unsigned num_units_in_tick, time_scale, fixed_frame_rate_flag;<br /> analyze_seq_parameter_set_data(num_units_in_tick, time_scale,<br /> fixed_frame_rate_flag);<br /> if (time_scale > 0 && num_units_in_tick > 0) {<br /> usingSource()->fFrameRate = time_scale<br /> / (2.0 * num_units_in_tick);<br /> } else {<br /> }<br /> break;<br /> }<br /> case 8: { // Picture parameter set<br /> // Save a copy of this NAL unit, in case the downstream object wants to see it:<br /> usingSource()->saveCopyOfPPS(fStartOfFrame + fOutputStartCodeSize,<br /> fTo - fStartOfFrame - fOutputStartCodeSize);<br /> }<br /> }</p> <p> //更新时间戳变量<br /> usingSource()->setPresentationTime();</p> <p> // If this NAL unit is a VCL NAL unit, we also scan the start of the next NAL unit, to determine whether this NAL unit<br /> // ends the current 'access unit'. We need this information to figure out when to increment "fPresentationTime".<br /> // (RTP streamers also need to know this in order to figure out whether or not to set the "M" bit.)<br /> Boolean thisNALUnitEndsAccessUnit = False; // until we learn otherwise<br /> if (haveSeenEOF()) {<br /> // There is no next NAL unit, so we assume that this one ends the current 'access unit':<br /> thisNALUnitEndsAccessUnit = True;<br /> } else {<br /> Boolean const isVCL = nal_unit_type <= 5 && nal_unit_type > 0; // Would need to include type 20 for SVC and MVC #####<br /> if (isVCL) {<br /> u_int32_t first4BytesOfNextNALUnit = test4Bytes();<br /> u_int8_t firstByteOfNextNALUnit = first4BytesOfNextNALUnit<br /> >> 24;<br /> u_int8_t next_nal_ref_idc = (firstByteOfNextNALUnit & 0x60)<br /> >> 5;<br /> u_int8_t next_nal_unit_type = firstByteOfNextNALUnit & 0x1F;<br /> if (next_nal_unit_type >= 6) {<br /> // The next NAL unit is not a VCL; therefore, we assume that this NAL unit ends the current 'access unit':<br /> thisNALUnitEndsAccessUnit = True;<br /> } else {<br /> // The next NAL unit is also a VLC. We need to examine it a little to figure out if it's a different 'access unit'.<br /> // (We use many of the criteria described in section 7.4.1.2.4 of the H.264 specification.)<br /> Boolean IdrPicFlag = nal_unit_type == 5;<br /> Boolean next_IdrPicFlag = next_nal_unit_type == 5;<br /> if (next_IdrPicFlag != IdrPicFlag) {<br /> // IdrPicFlag differs in value<br /> thisNALUnitEndsAccessUnit = True;<br /> } else if (next_nal_ref_idc != nal_ref_idc<br /> && next_nal_ref_idc * nal_ref_idc == 0) {<br /> // nal_ref_idc differs in value with one of the nal_ref_idc values being equal to 0<br /> thisNALUnitEndsAccessUnit = True;<br /> } else if ((nal_unit_type == 1 || nal_unit_type == 2<br /> || nal_unit_type == 5)<br /> && (next_nal_unit_type == 1<br /> || next_nal_unit_type == 2<br /> || next_nal_unit_type == 5)) {<br /> // Both this and the next NAL units begin with a "slice_header".<br /> // Parse this (for each), to get parameters that we can compare:</p> <p> // Current NAL unit's "slice_header":<br /> unsigned frame_num, pic_parameter_set_id, idr_pic_id;<br /> Boolean field_pic_flag, bottom_field_flag;<br /> analyze_slice_header(<br /> fStartOfFrame + fOutputStartCodeSize, fTo,<br /> nal_unit_type, frame_num, pic_parameter_set_id,<br /> idr_pic_id, field_pic_flag, bottom_field_flag);</p> <p> // Next NAL unit's "slice_header":<br /> u_int8_t next_slice_header[NUM_NEXT_SLICE_HEADER_BYTES_TO_ANALYZE];<br /> testBytes(next_slice_header, sizeof next_slice_header);<br /> unsigned next_frame_num, next_pic_parameter_set_id,<br /> next_idr_pic_id;<br /> Boolean next_field_pic_flag, next_bottom_field_flag;<br /> analyze_slice_header(next_slice_header,<br /> &next_slice_header[sizeof next_slice_header],<br /> next_nal_unit_type, next_frame_num,<br /> next_pic_parameter_set_id, next_idr_pic_id,<br /> next_field_pic_flag, next_bottom_field_flag);</p> <p> if (next_frame_num != frame_num) {<br /> // frame_num differs in value<br /> thisNALUnitEndsAccessUnit = True;<br /> } else if (next_pic_parameter_set_id<br /> != pic_parameter_set_id) {<br /> // pic_parameter_set_id differs in value<br /> thisNALUnitEndsAccessUnit = True;<br /> } else if (next_field_pic_flag != field_pic_flag) {<br /> // field_pic_flag differs in value<br /> thisNALUnitEndsAccessUnit = True;<br /> } else if (next_bottom_field_flag<br /> != bottom_field_flag) {<br /> // bottom_field_flag differs in value<br /> thisNALUnitEndsAccessUnit = True;<br /> } else if (next_IdrPicFlag == 1<br /> && next_idr_pic_id != idr_pic_id) {<br /> // IdrPicFlag is equal to 1 for both and idr_pic_id differs in value<br /> // Note: We already know that IdrPicFlag is the same for both.<br /> thisNALUnitEndsAccessUnit = True;<br /> }<br /> }<br /> }<br /> }<br /> }</p> <p> //注意！注意！注意！此处计算时间戳！！<br /> if (thisNALUnitEndsAccessUnit) {<br /> usingSource()->fPictureEndMarker = True;<br /> ++usingSource()->fPictureCount;</p> <p> // Note that the presentation time for the next NAL unit will be different:<br /> struct timeval& nextPT = usingSource()->fNextPresentationTime; // alias<br /> nextPT = usingSource()->fPresentationTime;<br /> double nextFraction = nextPT.tv_usec / 1000000.0<br /> + 1 / usingSource()->fFrameRate;<br /> unsigned nextSecsIncrement = (long) nextFraction;<br /> nextPT.tv_sec += (long) nextSecsIncrement;<br /> nextPT.tv_usec = (long) ((nextFraction - nextSecsIncrement)<br /> * 1000000);<br /> }<br /> setParseState();</p> <p> return curFrameSize();<br /> } catch (int /*e*/) {<br /> return 0; // the parsing got interrupted<br /> }<br /> }

每当开始一个新帧时，计算新的时间戳．时间戳保存在fNextPresentationTime中，在usingSource()->setPresentationTime()中传给fPresentationTime.
哇，我们看到live555的类之间调用关系曲折复杂，的确有点不易维护啊！同时我写的也不够清析，自己看着都晕，如果把你搞晕了，这很正常哦！

fPresentationTime是64位的时间，经convertToRTPTimestamp转换为32的rtp时间戳，见函数：

u_int32_t RTPSink::convertToRTPTimestamp(struct timeval tv)<br /> {<br /> // Begin by converting from "struct timeval" units to RTP timestamp units:<br /> u_int32_t timestampIncrement = (fTimestampFrequency * tv.tv_sec);<br /> timestampIncrement += (u_int32_t)(<br /> (2.0 * fTimestampFrequency * tv.tv_usec + 1000000.0) / 2000000);<br /> // note: rounding</p> <p> // Then add this to our 'timestamp base':<br /> if (fNextTimestampHasBeenPreset) {<br /> // Make the returned timestamp the same as the current "fTimestampBase",<br /> // so that timestamps begin with the value that was previously preset:<br /> fTimestampBase -= timestampIncrement;<br /> fNextTimestampHasBeenPreset = False;<br /> }</p> <p> u_int32_t const rtpTimestamp = fTimestampBase + timestampIncrement;</p> <p> return rtpTimestamp;<br /> }其实时间戳的转换主要就是把以秒为单位的时间，提升成按频率为单位的时间．也就是提升后，时间间隔不是以秒为单位，而是以1/fTimestampFrequency为单位，也就是1/9000秒。然后再强转为32。