Is there a list of the required tags for DNG? The terminology in the specification is inconsistent and not defined. If a tag has a default value are readers required to use the default value even if the tag isn't present? Is a default value of "None" the same as "Required tag"? Why are there Adobe implementation specific tags in the specification instead of leaving them in a private IFD?

The specification says that a DNG should follow "all of the formatting rules of a TIFF". What does that mean exactly? I thought it meant the structure section of the TIFF spec, but the dng_validate code allows a magic number other than 42:


case 42: // Standard TIFF value
case 85: // Panasonic
case 0x4f52: // Olympus
case 0x5352: // Olympus


The code seems to indicate a magic number is required, but the specification doesn't say what it should be... How come Panasonic and Olympus have magic numbers and not other companies? How can I get my own magic number for my application? Which parts exactly of the TIFF specification are DNG files required to comply with?

I'm getting frustrated with the DNG specification's lack of specificity. The document reads more like internal software documentation than a specification. I shouldn't have to step through example code to figure out exactly how to write a format that wants to be a standard.

For interest, are you using the TIFF/EP specification (ISO 12234-2)? The DNG specification says:

"DNG is an extension of the TIFF 6.0 format, and is compatible with the
TIFF-EP standard. It is possible (but not required) for a DNG file to
simultaneously comply with both the Digital Negative specification and
the TIFF-EP standard."




A quote that may be relevant:

"TIFF/EP tag definitions do not allow default values. All values shall
be explicitly stated".




And a specification of bytes 2 & 3:

"An arbitrary but carefully chosen number (42) that further identifies
the file as a TIFF file".

If a tag has a default value are readers required to use the default value
even if the tag isn't present?




The meaning of "default value" is, that it is to be used in case there is no explicite specification.

In other words, the default value will be used *only* when the tag is not present.

Why are there Adobe implementation specific tags in the specification
instead of leaving them in a private IFD?




If the question is, why the tags specific to raw images are in the image's IFD, well, that IFD describes the image, so it is natural that they are there. They could not be "left" in a "private" IFD, because they were not there before.

the dng_validate code allows a magic number other than 42:




I guess it is always hard on the developers of some standard to support deviations, which are actually violations of the standard, but if one sticks to the principle that these deviations be rejected, then the user basis gets smaller.

Violations of the standard are rampant; the software developers of some camera manufacturers don't give a damn for the standards. For example the tags within an IFD are supposed to be in the ascending order of their code. Well, don't count on that. This is not the question of pure syntax: if the sequence were strict, when processing some particular tag one could count on the fact, that a related, preceding tag has been processed already or the default value has to be used. No, you have to delay the processing of interrelated tags.

I have only two beefs with the DNG specification:

1. (this is not a syntactical issue): there is only a single white level value, although pixels with different colored filters should have different values,

2. the DNGPrivateData tag is a joke. Its specification states:

The private data must be byte-order independent. If a DNG file is converted from a bigendian file to a little-endian file, the data must remain valid

and then you take a look at the actual data: it starts with "Adobe MakN" followed by the length of the data in MSBF byte order, despite that the TIF in this particular case is otherwise in LSBF format, and that the length field is immediately followed by "II", indicating that the following data too is in LSBF byte order.

However, once you figured it out, it's ok.

"but the dng_validate code allows a magic number other than 42".





"I guess it is always hard on the developers of some standard to support
deviations, which are actually violations of the standard, but if one
sticks to the principle that these deviations be rejected, then the user
basis gets smaller".




I just took an otherwise valid DNG file, and, using a hex editor, changed the magic number to 85, one of those mentioned by Sue Dunham.

Then I ran dng_validate. (The output text says: "Magic number = 85"). It output an error message to the command stream saying: "*** Error: Invalid TIFF magic number ***". When I tried to open it, ACR said "Cannot complete your request because it is not the right kind of document".

Is there known to be - in practice - ANY value other than 42 that WON'T cause a command stream error message to be output, and that WILL be opened by ACR? I suspect not.

For interest, I just ran dng_validate on a Panasonic FZ8 raw file file. The error message was "Error: Missing or invalid DNGVersion ***". (The output text says: "Magic number = 85").

I think dng_validate is being a bit tolerant in order to produce useful output when analysing some non-DNG files, but that doesn't mean it suggests that they are valid. (I sometimes use dng_validate to have a look inside various other files, but I always have to ignore its failure messages - it doesn't say they are valid DNGs).

(For completeness, I changed the 85 in the Panasonic FZ8 raw file to 42, and tried again. Same failure message from dng_validate. Photoshop opened it as a "Photoshop Raw" file, which is a little-known format that is TOTALLY different from camera raw files, and bypassed ACR entirely).

"it starts with "Adobe MakN" followed by the length of the data in MSBF
byte order, despite that the TIF in this particular case is otherwise
in LSBF format".




So what?

Isn't the specification saying that you must be able to treat the DNGPrivateData as a string of bytes, without having to make ANY change to them when changing the order of the overall DNG file?

In which case, surely the order of the bytes in the length WITHIN the DNGPrivateData HAS to be independent of whether the DNG is bigendian or little-endian? If the byte-order in the length changed when the byte-order of the DNG file changed, surely it would then be necessary for whatever software changed the order of the DNG file to know the internal structure of this particular DNGPrivateData so that it could change the order of this length field? (Which would defeat the purpose).

Surely Adobe had to decide which way round this value was going to be stored, and stick to it whatever the byte-order of the DNG file? (Would you have been happier if it had stored the length as a string of decimal characters?)

Isn't the specification saying that you must be able to treat the DNGPrivateData
as a string of bytes, without having to make ANY change to them when changing
the order of the overall DNG file?




No, it is not (it would be a colossal nonsense).

Btw, since when has a 32 bit long binary number been a "string of bytes", no matter in which byte order? It would have to be in ASCII form in order to be interpreted independently of the byte order (and thta fact would have to be documented).

In which case, surely the order of the bytes in the length WITHIN the
DNGPrivateData HAS to be independent of whether the DNG is bigendian or
little-endian?




Apart from the fact, that this is not the case,

there is no such thing as independency from the byte order as long as there are binary numbers with more than one bytes in the data

and there are, lots. However, that's not the point, because

the length is OUTSIDE of the quoted data.

The specification might state, that the length is always in MSBF format, but it does not.

Would you have been happier if it had stored the length as a string of
decimal characters?




I would be happier, if the length were stored in the byte order of the actual file, because this header is still in the domain of DNG, not of the quoted data.

I think the only solution now is to state in the DNG specification in cleartext, that the length is always in MSBF form, notwithstanding the native byte order of the file. This will sound ridiculous in light of the high-flying nonsense "the private data must be byte-order independent", but at least it would document the actual structure of the DNG file.

"No, it is not (it would be a colossal nonsense)".




Why is that nonsense? Surely Adobe are trying to ensure that products that manipulate DNGs, for example moving stuff around within it by changing previews or changing other metadata, can do so safely? They don't have to worry about damaging the DNGPrivateData, even if they don't understand it.

So it needs to be possible to copy the DNGPrivateDate without any change, even if the new version of the file change changed its byte-order. (And the same should ideally apply if the private data remains in the Makernote, which is the reason for MakerNoteSafety).

"I would be happier, if the length were stored in the byte order of the
actual file, because this header is still in the domain of DNG, not of
the quoted data".




Not true! There is no need to have that field in DNGPrivateData if the DNG writer doesn't want it there. It is simply Adobe's "private" way of storing the original Makernote in the results of a DNG conversion. It is NOT part of the specification that anyone else needs to be aware of! Adobe has to obey its own rules for the whole of the DNGPrivateData string.

Therefore, any product that changes the DNG file, and changes the byte order as a result, has no reason to believe there is field in the DNGPrivateData that can't just be copied across, but has to have its byte order reversed. After all, a camera that writes to DNGPrivateData doesn't have to put such a field into the DNGPrivateData.

For interest, here is the start of DNGPrivateData for a Pentax K10D PEF converted DNG via the DNG Converter:
41 64 6f 62 65 00 4d 61 6b 4e 00 01 30 06 4d 4d
00 00 02 ac 41 4f 43 00 4d 4d 00 5b 00 00 00 01
00 00 00 04 03 00 00 00 00 01 00 03 00 00 00 01
00 00 00 00 00 02 00 03 00 00 00 02 02 80 01 e0

And here is the of DNGPrivateData for a Pentax K10D's native DNG (no DNG Converter):
50 45 4e 54 41 58 20 00 4d 4d 00 5a 00 00 00 01
00 00 00 04 03 00 00 00 00 01 00 03 00 00 00 01
00 00 00 00 00 02 00 03 00 00 00 02 02 80 01 e0
00 03 00 04 00 00 00 01 00 00 66 bf 00 04 00 04

After the null-terminated string at the start ("Adobe" or "PENTAX ") there is no need for them to be similar. (I believe the Pentax Makernote content starts just after the 2nd "MM" in the first case, and just after the first "MM" in the second case - Pentax supplied those particular "MM"s).

Only the null-terminated string at the start is part of the formal DNG specification. The rest is up to the original DNG writer.

First, let's review how it *is* now.

The DNG specification does not state, that "you must be able to treat the DNGPrivateData as a string of bytes", and that's very good so.

If my goal would be to only copy the private data around, I would throw it away. I don' keep data only to be able to copy it.

If I keep the private data, the reason is, that I want to process it, which requires that I can interpret it.

Now, the private data is a mess and I am not expecting Adobe's DNG converter to sort out the what the camera manufacturers are creating. All I am expecting is, that it won't become more messy than it was in the original raw file.

The DNG specification requests:

The private data must be byte-order independent. If a DNG file is converted from a bigendian file to a little-endian file, the data must remain valid

This is, of course nonsense, for

1. there is no such thing as byte order independency with binary numbers,

2. the camera manufacturers don't give a damn for Adobe's "must".

However, this is only a documentation issue. It should say something like

The private data must be interpreted on it's own, independently of the byte order of the DNG file.

The point is namely, that if the byte order of the DNG file changes, the byte order of the encapsulated data will not be changed.

Anyway, I was not complaining about the choise of words in the DNG specification.

Now, the other side.

Adobe can not force the camera manufacturers to adhere any requirements within their respective MakersNote, but the DNGPrivateData is purely DNG specific, so the DNG specification could have been more specific in the generic structure of this tag.

However, if the DNG specification does not specify the structure behind the identification string, then it is the responsibility of the creators of the particular DNGPrivateData to describe it's structure. Adobe's DNG converter is one of those, and I find it only natural, that the structure of the data will be documented (I saw already an incorrect interpretation of it in this forum).

Back to my complain: it is more than strange to embed an MSBF field in a LSBF file, indicating the length of an LSBF field.

"The DNG specification does not state, that "you must be able to treat
the DNGPrivateData as a string of bytes", and that's very good so. If
my goal would be to only copy the private data around, I would throw it
away. I don' keep data only to be able to copy it".




The REQUIREMENT is that "you must be able to treat the DNGPrivateData as a string of bytes" if you choose! YOU PERSONALLY may want to process the private data, but that is NOT the case for every product that handles DNGs. A product that changes the XMP metadata, or changes the preview sizes, (etc), needs to be able to change the positions of nearly all the data in the resultant DNG file, and even change the file's byte-order, without ANY need to do ANYTHING with the DNGPrivateData other than to copy it around without understanding its content. That is why it is specified as it is.

Note - the only part of the DNGPrivateData that is part of the DNG specification is the null-terminated string at the start that identifies the company that (in effect) owns the specification for the DNGPrivateData following the null-termination. There is no rule that anyone (other than that company) should know what the rest of the format is, or be able to process it. There is actually no rule that it must not be encrypted - that is a PRIVATE matter for that company (which is why it is called DNGPrivateData!) All that is required is that the company (internally) designs it in such a way that it still makes sufficient sense TO THAT COMPANY if it is moved around and the byte order of the file is changed by someone else.

The data immediately after the initial null-terminated string "Adobe" is specific to Adobe, and NOT part of the DNG specification. This is (I think) 14 bytes long. There is no rule that anyone else needs to be aware of the format of those bytes, and certainly no rule that they must be manipulated by any non-Adobe product. Adobe has published it (in the SDK) in case any product wants to process the rest of the DNGPrivateData following those 14 bytes - they are the original EXIF Makernote (and not necessarily understood by Adobe). Those 14 bytes bytes are unlikely to appear in any DNG created by a company other than Adobe.

"Now, the private data is a mess and I am not expecting Adobe's DNG converter
to sort out the what the camera manufacturers are creating. All I am expecting
is, that it won't become more messy than it was in the original raw file".




All the DNG Converter does is generate the initial null-terminated string, then the 14 bytes (or whatever) that identifies the original context for the Makernote, then copies across the original Makernote unchanged. It is unfortunate that this looks more complicated than the original Makernote, but how else can Adobe ensure that other products can move the DNGPrivateData around without corrupting the original Makernote beyond analysis?

"The DNG specification requests: The private data must be byte-order independent.
If a DNG file is converted from a bigendian file to a little-endian file,
the data must remain valid. This is, of course nonsense, for"




That is SENSIBLE and indeed MANDATORY if DNG is to meet its objectives. It certainly isn't nonsense! See above - it is a REQUIREMENT that DNGPrivateData can be handled that way!

"1. there is no such thing as byte order independency with binary numbers,"




Nonsense! All you are saying is that it can't be done directly using the primitive instructions of the processor, because Intel-oriented and Motorola-oriented processors work differently. But it is obviously trivial to treat a multi-byte value as an integer larger that 255 in a consistent way by cycling through the bytes one-by-one! I assume that is how the values within the Adobe-defined 14 byte field are to be handled?

"2. the camera manufacturers don't give a damn for Adobe's "must"."




Then they will create DNGs that get corrupted by other products! It is their choice - obey the rules and succeed, break the rules and fail. The point is, those other products WILL move DNGPrivateData around, and perhaps put it into files with a different byte order, and the camera manufacturers can't negotiate that!

"Adobe can not force the camera manufacturers to adhere any requirements
within their respective MakersNote, but the DNGPrivateData is purely DNG
specific, so the DNG specification could have been more specific in the
generic structure of this tag".




It is PRIVATE data! The whole point is NOT to be any more specific than is needed to satisfy the objectives for handling DNG files.

"However, if the DNG specification does not specify the structure behind
the identification string, then it is the responsibility of the creators
of the particular DNGPrivateData to describe it's structure. Adobe's DNG
converter is one of those, and I find it only natural, that the structure
of the data will be documented (I saw already an incorrect interpretation
of it in this forum)".




No, the creators of the particular DNGPrivateData have NO such responsibility. It is PRIVATE data, hence its name! No one outside the company concerned has a right to know anything about the format of what follows the initial null-terminated string.

Adobe is willing for other companies to understand (and exploit) its 14 bytes. That is not a precedent that any other company should do this.

"Back to my complain: it is more than strange to embed an MSBF field in
a LSBF file, indicating the length of an LSBF field".




Given the REQUIREMENT that the whole DNGPrivateData is byte-order independent, that is unremarkable. It can be handled.