DM2E

[kentaro-k.21]

this is a tool to extract few images from DM2 graphics.dat.
source codes are not contained for now. they are posted in few weeks.
I wrote in C# and .net framework required to run.
thanks.

Kentaro-k.21

[Gambit37]

Wow! I've just had a play around with this tool and it's a great step forward in DM2 graphics extraction. The biggest step is that you've managed to get the palette mapping more or less correct. In fact it may be completely correct, but I'm unsure until I do a bit more research. Great stuff!

Obviously, there's still the massive task of interpreting the format of the remaining BIN files, but setting that aside here's some of the things I noticed about the graphics.

1) Some images are offset, so that they appear wrapped around an arbitrary midpoint, which means the right of the image is on the left, and vice versa. It's easy to fix this using something like Photoshop to manipulate the image, but it would be great if we can extract them correctly in the first instance.

2) At first I thought the transparency was messed up, but then I realised how clever FTL have been about using as little space as possible to store animation changes. If you look at the Thorn Demon from the Amiga version, it only has 4 complete frames, 7 other frames (that appear to be messed up) and two masks. However, I took all this lot into PhotoShop, and using layers and making assumptions about which colours are transparent, I was able to build up all the animation frames by overlaying these different images, and using the masks to block out areas that aren't required. It's very clever! It's not perfect which either suggests that the extraction process is not 100% or my manipulations are a little wrong, but it's pretty close.

Here's an image to show you what I mean:
http://homepage.ntlworld.com/matt_hill/thorndemon.gif

[Gambit37]

I just took a quick play of Amiga DM2 under WinUAE and took a screengrab. The palette is pretty close, but it's not 100% correct yet, as you can see from this comparison:

http://homepage.ntlworld.com/matt_hill/ ... aldiff.gif

Is there anything useful I can do with palette research to help this tool along??? I'm pretty good at stuff like this, having pixel edited graphics since my deluxe paint Amiga days, and I know a lot about palettes and colour spaces.

[kentaro-k.21]

hi, Gambit.
thanks for your patience and support!

i confirmed about palette difference issue.
you're right. palettes seem not to be identical with your Amiga screen shot.
my palettes are completely grabbed from graphreadsource and they're unmodified.
i'll have to adjust them once.
and i'm glad to have offer of your palette/color knowledge. what kind of information would you want to offer me.
thanks.

i documented what i do with current uncompression process.
the document is html format and will be attached with this post.

Kentaro-k.21

[kentaro-k.21]

i corrected some mistake of the prior document and added a little bit description: Technical Documentation - Dungeon Master II Graphics.dat: How to read some messed up images http://dmweb.free.fr/?q=node/1059

from history part,

Rev.2
- Corrects for some miscalculation
- Detailed description for boundary between Local palette and compressed image.

[kentaro-k.21]

hi.
i created another extraction tool for dm2.
this one generates some htmls and extracts much images.
it will recognize what kind of data it is and arrange elements into each html file.
and... this program is written in C# too.
and for now this archive has no readme nor document. sorry

i'll report the new progress when i have much time to write documents
thanks

Kentaro-k.21

[ChristopheF]

You're doing a fantastic job, Kentaro!!
Keep up the good work, and thanks for the format documentation!

Note: DM2AE worked fine for me with the amiga graphics.dat, but with the PC graphics.dat, a lot of images (most of them) were not correctly decoded.

Christophe

[Gambit37]

This is very good work, the second release is another step forward.

The images for the Amiga version are now complete, but they still do not have the correct colours. Where do you get the palette information from? Because it seems that the Amiga images are actually using palette information from the PC version...

Don't forget that the palettes in DM (both Amiga and PC) use only 64 colour registers to mix their colours. Therefore:

White: RGB 63 63 63
Red RGB 63 0 0
etc...

When these palettes are converted to Windows, each register must be multiplied by 4. So the maximum setting can be:

White: RGB 252 252 252

If there are any colours registers that end in an ODD NUMBER after conversion, then you know that the colour IS WRONG.

Have you decoded the palette from the GRAPHICS.DAT file? How do you get the correct colours for exach pixel? Is each pixel given a colour, or does it use a colour map to pick the correct colour from the palette?

I don't know if this helps you at all....???

[ChristopheF]

Gambit, have a look at the P4B1 data type on the Data Files format page.

Color palettes in DM and CSB are defined with RGB components varying from 0 to 7, so:
White: 7 7 7
Red: 7 0 0
etc...

Each color definition uses 9 bits (stored in a word).

I believe all the color palettes are defined in one of the RAW1 files (have a look at CSBwin source code for the exact location). I never took time yet to decode them and implement the correct palettes in my graphics extraction script.

The previous information is for the Atari ST versions of both DM and CSB, and I believe all other ports use the same color palettes, although depending on the platform, they may be defined in larger color spaces (on PC in VGA mode, they probably are defined as you said, with color values ranging from 0 to 63).

Christophe

[Gambit37]

I must admit I was going on memory rather than actual raw info.

The data about colour spaces using 64 registers for the PC VGA palettes is correct though.

[Gambit37]

I've had another look through the images for Amiga DM2, and discovered quite a few that aren't right. There are some images that have been combined incorrectly so they have bits of other frames attached.

I'll make a list and upload them.

I've also noticed that all the other interface graphics are missing (such as the save/load dialogues)... these were present when extracted using version 1.... can you put them back in???

[kentaro-k.21]

hi.
thanks for a lot of reply. i'm just amazing.

to Christophe:
thanks! i'll try to decode a little bit more and publish it.
i'm currently trying to decode the images which is encoded in another format on dm2 pc version, they are extracted in clearly messed up status.
the other day, i could know some rules in them. but it's not enough to decode an image.
i must inspect many images and find the complete rules...
if i find an intelligence about it, i'll report later.

to Gambit:
i confirmed and repaired the problem about images are missing from extraction.
they are referenced from other category which i couldn't name it then, and finally i forgot the code for them. sorry.
next version will extract them as unnamed categories.

and about corruption of extracted image, i could confirm about them and repaired the problem.
it'll be found at compass images which are extracted from amiga dm2 graphics.dat.
misextraction came from bug of decoder part. it is that i completely rewrote the decoder for DM2AE and it just had a bug.
though i repaired this bug, there is another misextraction found on mummy images. some images seem to have two mask colors.
i'll check it later...

thanks for your support!

[kentaro-k.21]

2 documents:
- Technical Documentation - Dungeon Master II Graphics.dat: How to read a directory entry http://dmweb.free.fr/?q=node/1060
- Technical Documentation - Dungeon Master II Graphics.dat: How to read the Text data http://dmweb.free.fr/?q=node/1057

[rain`]

About the directory entry information:

Why is there over 10000 entries in the "directory file" (first extracted file), but yet only around 5000 files in the graphics.dat file?

I've been tryin to figure that out, but im stumped!

[kentaro-k.21]

hi.

to Christophe:
i don't mind about making adaptation of the research result i published, so please complete and update your documentation with it.
it will be good resource for tool/editor developer to create other cool dm2 tools.
thanks

to rain`:
it's a good question.
i'm also interested in your issue, so i just made a statistics table with excel.
http://dmweb.free.fr/Stuff/DM2E/DMII-Mi ... cs_tbl.rar

the table shows how many reference there are from directory entry.
it assumes that 7th & 8th byte in each file entry are reference indices to point a binary in graphics.dat.
so, some file entry may point invalid binary index which is not existed. they're also counted in other column.

[kentaro-k.21]

hi.
here is update of DM2AE.
in this version, some messed up images (the extraction result of prior version) in DM2 PC version will be extracted correctly.
in this release, archive lacks the tutorial and source codes. they'll be back in next stable version.
thanks

[kentaro-k.21]

hi.
i could document about dm2 pc compressed image format to html page: Technical Documentation - Dungeon Master II Graphics.dat: How to read and uncompress images from the PC version http://dmweb.free.fr/?q=node/1058.
also a dm2 pc palette as png file:
http://dmweb.free.fr/Stuff/DM2E/DMII-Misc-Dm2pcpal.png
i could obtain the palette from dm2 pc through vmware.

thanks.

[kentaro-k.21]

hi.
this DM2AE update contains some minor changes/improvements, please check it.

changes from prior version:
- extract pcm sound as wav files. --- i referenced the File Decoder vbs code to export pcm wav file. very thanks for Christophe and his great work the Data File Decoder!
- extract png or gif as 8-bit indexed palette bitmap image.
- partial correct for misextraction on some small images (e.g. blood-key hole)
- a little of item information will be extracted and included in html. (they are weight, money value, food value, portability, equipment bonus value)
- each categorized html page has index on top
current known limitation and remained problems/subjects:
- no tutorial included due to "in develop" version
- wrong palette color is assigned to some extracted images from dm2 amiga (reported by Gambit)
- exporting images with transparent color key

[Gambit37]

BRILLIANT! FANTASTIC WORK! Thank you Kentaro! Your extraction tool is essentially 100% complete. I extracted the graphics from DM2 PC and Amiga and couldn't see any problems with the images. The colours look correct, and all the elements appear to be in the correct place. You must have spent a lot of time on this.

I've been waiting for a tool like this for ages so that I can work on the DM2 part of my website. It's no fun trying to screencapture all the creatures then paint out the bits you don't want....

Thank you!

[Gambit37]

I've just checked your the palette that your using; it's slightly wrong. I'm not sure how you created it, but some of the indexes used RGB values that do not end in even numbers.

I've uploaded the CORRECT palette for DM2 PC; I've included a BMP screencapture from DM2 running under the correct 8bit VGA mode, and I've also saved the palette as Microsoft and Paint Shop Pro formats. I've also included versions of your palette so that you can see the difference.

http://dmweb.free.fr/Stuff/DM2E/DMII-Mi ... lettes.rar

Maybe this will help you get the colour mappings 100% correct?

[ChristopheF]

Gambit is right: you've accomplished a great task!

I will use parts of the extracted data to complete my site with new pages about DMII and maybe a new skin.

Thanks a lot!

Two questions:
Do you know if you've decoded all the items, or are there still a few that you can't figure out? ...containing color palette(s) for example

Again about color palette(s): are you absolutely sure that DMII for PC uses only a single 256 color palette? maybe there are several palettes?
What about the amiga version which has only 16 colors images?

Christophe

[Gambit37]

I'm fairly certain that DM2 for PC uses only one palette, although I believe that it uses some kind of mapping table to produce effects like the "super brightness" when fireballs explode or the lightning strikes.

I think DM2 Amiga must use a combination of palettes, probably 32 colours each. You can see this quite clearly when you exit the first room and there is a Glop in the corridor. Try and watch what happens to the colour of the Glop's eyes as it moves between the boundary separating the stone corridor from the glade outside... They also change as the party crosses the boundary, which suggests that there are distinct palettes used for different areas of the game. I'm not 100% sure about that though...

[kentaro-k.21]

hi Christophe, Gambit!
i'm very interested in your questions, i tried to figure out the answer yesterday.

about progress of extraction, it's difficult question.
if it means whether unknown binary data in graphics.dat of dm2 pc still exists or not, 200 binary data (out of 8030) seem to be still unknown (including hmimidi data).
if it means whether unknown entries in dir entry still exists or not, ... now investing.
i'm trying to make a reference table to figure out what is unknown and what is known. i'll upload the statistics as excel tables in few days.

about dm2 pc palette, i believe that there is single static palette in game. DM2AE also uses the same palette to decode title screen and staff roll screen image as other images have.
and about the opinion "super brightness" by Gambit, i agree with him and it may be sure with attached screen shots.
i replaced the portait image of tempest with color palette image (16 by 16, 256 pixels. each pixel assigned with each color).
if any color in palette is changed while "super brightness" is invoking, the portlait image also will be affected.
http://dmweb.free.fr/Stuff/DM2E/DMII-Mi ... Normal.png
http://dmweb.free.fr/Stuff/DM2E/DMII-Mi ... Bright.png
at last, palette has not changed.

about dm2 amiga palette, i don't know anything. i'm not good at amiga machine, so i don't know even how many colors are simultaneously realized in any graphics mode.
if there is any plan to inspect dm2 amiga palette, i will be able to help you by modifying images in graphics.dat.

i checked your palettes, Gamit!
i also understood about the palette condition of SVGA mode with your past reply: it has 6 bit register in each color element to combine R, G, B elements into a color.
therefore your concluded the white color is assigned like "White: RGB 63 63 63"
in windows, full or true color mode seems to have 8 bit register in each color element.
now when the 6 bit register color (from 0 to 63) is converted into 8 bit register color space (from 0 to 255), the brightest intensity, 252 is figured out as 63 * 4 = 252.
but i have dissent. i think White color should be converted to 255,255,255.
i want your opinion about color space conversion, please feel free to comment!

[Tom Hatfield]

Kentaro: Multiply the 6-bit RGB values by 4.047619 (this is the same as dividing by 63 and then multiplying by 255) and truncate the decimals. This will give you a value between 0 and 255, which is what you want.

[Tom Hatfield]

By the way, your utility is completely awesome. I really love the way you set it up to be viewed through web pages. Reminds me of the ngStats logging in Unreal Tournament. You've done outstanding work.

[kentaro-k.21]

http://dmweb.free.fr/Stuff/DM2E/DMII-Mi ... bl_DM2.rar
the archive contains 3 csv files and 1 excel basic module file.
each csv file describes about use of binary data and references from directory entry in each graphics.dat file (amiga, pc, pc9801 versions).
unknown binary data are marked as "Raw".
DecorateSheet.bas will arrange the excel sheet so that you can see visually.

how to apply the effect of DecorateSheet.bas with Excel 2000:
- opens the csv file
- presses [Alt]+[F11] key to show Microsoft Visual Basic window.
- presses [Ctrl]+[M] key to import basic module file. selects the DecorateSheet.bas.
- presses [F5] key to execute the published macro program. selects DecorateSheet macro and runs. it takes a minute.

[ChristopheF]

Kentaro-k.21, here is some information and changes I'd like to see in the next version of DM2AE (if you still plan to work on it):

1) Can you rename the following catagory names in HTML pages so that they are more meaningful:
Category 1 -> "Interface - Main Screen"
Category 3 -> "Messages"
Category 5 -> "Main Menu and Credits"
Category 6 -> "Credits"
Category 7 -> "Interface - Character Sheet"
Structural -> "Dungeon Graphics"
Category 9 -> "Wall Ornates"
Category 10 -> "Floor Ornates"
Category 11 -> "Door Ornates"
Category 12 -> "Missiles"
Category 24 -> "Teleporter"
Category 25 -> "Skullkeep and Weather"
Category 26 -> "Dialog boxes"

2) Can you rename the data type called "0B" to "Word" (2 bytes values). (like "image", "text", "sound")

3) Based on item information from the DM2 Adventurer's Guide available on this site and my tests with transparency, I have identified the following data values (PC version, I don't know for other versions):

Creatures
0B - 04 00 00 - Index of transparent color
Clothes
0B - 0B 00 00 - First byte: Sharp resistance, Second byte: Armor strength
Weapons
0B - 08 00 00 - Attack strength

Could you display these values with their description in HTML pages?

4) Question
The transparent color is specified for creatures images. But for all other images, how DM2 can know which color is transparent in each image?
It is not always the same color index in all images... Maybe the unknown data not decoded yet contains information about transparency?

5) If we can find all transparency information, I'd like to see the .PNG files to include the transparency information (or make it optional).
You could already do that now for creatures.

Christophe

[ChristopheF]

Some more info about weapons:

Weapons
0B - 08 00 00 - Attack strength
0B - 09 00 00 - Throw strength
0B - 0A 00 00 - Accuracy (for missile weapons)

Christophe

[kentaro-k.21]

hi, Christophe
i'll update DM2AE and publish in about a week with your discovered information. thanks for your support and information!

my personal oppinion(answer) for each suggestion:

1) yes! it will work on next version.

2) hmm.. frankly speaking, this is difficult problem. i noticed that there was another type(one is "0C") to have word value like "0B". i omitted them from extraction because then i couldn't distinguish that they have a word value or a binary index.
next table shows the known info(in my knowledge) of entry type. the word value meaning, binary index or simple word value, is decided whether it has any out of range binary index or not.
it says also there are still many unknown raw data types. and i had ignored and hid them because they are simply unknown and i think that no one may have interest for them.

entry types:
00 : ?? too few entries
01 : (binary index) image
02 : (binary index) sound
03 : (binary index) music (hmi midi? for DM2PC)
04 : (binary index) ?? unknown purpose
05 : (binary index) text
06 : (binary index) ?? unknown purpose
07 : (binary index) ?? unknown purpose
08 : (binary index) ?? unknown purpose
09 : (binary index) ?? unknown purpose
0A : unused
0B : (word value) miscellaneous value descriptor (equipment bonus, and so on)
0C : (word value) ?? unknown purpose
0D : ?? too few entries
0E : (binary index) ?? unknown purpose

anybody has any idea to show them as unknown data value (simply word value, extract as .BIN file and connect by link, binary dump, ...) or other form?

3) ok, i will make it works. thanks a lot!

4) at first, about DM2 compressed image; the format is similar with IMG3 and IMG4 data format used on DM1(and CSB?). some DM2 compressed image uses the extended format of IMG3/IMG4 (i often call newer format as ExtDM1). ExtDM1 format has local 16 colors palette and is attached at suffix of binary in 16 bytes. so, each image can select 16 colors from DM2 system palette (256 colors).
on my thought, index of transparent color may use the local color palette index to select transparent color.

and... at least DM2PC has another compressed image format. in that format, each pixel has 8 bit color space (1 of 256 colors) and is mapped into DM2 system palette directly, so no local palette information exists. thus, if transparent color index is 10, the 10th(0 based) palette color becomes transparent color.

IMG3/IMG4 format from: dmweb.free.fr/FFData.htm

5) in my research, i knew that .NET framework could generate .GIF file with transparent color. transparent color support may be adapted to .GIF file too.
in next version, transparent support may be added as experimental step.

recently, my working on DM2 is researching about actuators in dungeon dat. editing them and test again and again... i made some test maps with discovered information. but my research result is still not enough to represent all dungeon techniques found in DM2.

[ChristopheF]

Thanks for your complete answer.

About question 2), I'd really like to see ALL the binary data that you can read in the graphics.dat file exposed on the HTML pages.
You can make this optional if you think it would add a lot of non readable data, useless for most users.

But I think it would be a valuable tool to try to discover the meaning of each value, by comparing different values for different objects of the same kind.
That is how I found the few values in my previous post.

So I suggest that you should just display all the raw binary data (not yet interpreted) as hexadecimal values, like you did for values of type "0B".

On the subject of DM2 actuators, I think you should share information with Sphenx, as he has already found a lot of them (he his implementing a DM2 dungeon editor similar to his DMBuild).

Christophe