|
Post by dshadoff on May 22, 2020 15:40:29 GMT
I know that this has been a topic of research for several people here (although not so much for myself), so I'll post it here in case anybody wants to look into whether it is useful to them.
|
|
touko
Punkic Cyborg
Posts: 106
|
Post by touko on May 22, 2020 21:21:42 GMT
You also have the new LZSA : github.com/emmanuel-marty/lzsaLike the lz4 it"s a on the fly decompressor, so really fast but not with the better compression ratio.
|
|
grahf
Deep Blooper
Posts: 19
Homebrew skills: hobby gamedev
Fave PCE Shooter: Download / Final Soldier
Fave PCE Game Overall: Bomberman '94
Fave PCE RPG: Kaze no Densetsu Xanadu II
Currently Playing: Akumajo Dracula X
|
Post by grahf on May 23, 2020 10:15:55 GMT
Maybe not directly related, but I've always wondered how HuVideo could have been improved with knowledge of current compression techniques.
|
|
|
Post by elmer on May 26, 2020 20:04:41 GMT
You also have the new LZSA : github.com/emmanuel-marty/lzsaLike the lz4 it"s a on the fly decompressor, so really fast but not with the better compression ratio. I suspect that you may have a typo in your response ... I have yet to find a case (for 8-bit and 16-bit console/computer data) where LZ4 compresses better than Emmanuel Marty's LZSA1, let-alone his LZSA2 which does much, much better. Here are optimized smaller-and-faster HuC6280 decompressors for the PC Engine to complement the standard-6502 ones of mine that are included in Emmanuel Marty's project on github. Attachments:lzsa-huc6280.zip (5.93 KB)
|
|
touko
Punkic Cyborg
Posts: 106
|
Post by touko on May 27, 2020 14:46:12 GMT
Yes but it's also much slower too,in fact it seems that LZSA has multiple schemes depending on what you want, faster decompression but low ratio, or the contrary . The code seems a bit more complex than the LZ4 one . It's possible
|
|
|
Post by elmer on May 27, 2020 17:21:06 GMT
Yes but it's also much slower too,in fact it seems that LZSA has multiple schemes depending on what you want, faster decompression but low ratio, or the contrary. Yes, LZSA2 is about 20% slower to decompress than LZSA1, but then there is always a tradeoff between compression ratio and speed, or else we'd all use RLE for speed, or deflate for compression ratio. The point is ... LZ4 compresses worse than LZSA1 and isn't any faster, and LZSA2 compresses much better than either (sometimes approaching aplib), and it is still fast-enough that you're unlikely to ever notice a difference. I went through the whole thing with some LZ4 fans in a compression thread on AtariAge. You can see an LZ4-vs-LZSA1 comparison here, and a size-test of various compressors here. Please note that I continued to improve both the LZSA1 and LZSA2 decompressors after those posts, and they're both a little smaller and faster. The code seems a bit more complex than the LZ4 one. Most of what looks like complexity in my LZSA decompressors is just optional code paths so that smaller-or-faster versions of the code are both in the same source file. You can look at Peter Ferrie's 6502 versions of the code in the LZSA distribution on gihub. They're easier to read, but decompress significantly slower than mine. He's the guy that wrote the 6502 LZ4 decompressor that you're probably looking at.
|
|
touko
Punkic Cyborg
Posts: 106
|
Post by touko on May 27, 2020 18:24:34 GMT
Thanks for all the information, very instructive .
|
|
|
Post by elmer on Apr 7, 2021 2:51:20 GMT
An interesting new compression format has been created by Einar Saukas, ZX0. It performs rather nicely in recent testing, and it seems to offer the excellent compression of aPLib, with a small and fairly fast decompressor. My own testing shows that as a format, it seems to usually compresses a little smaller than aPLib, and decompresses about 10% faster than the "enhanced" 6502-friendly aPLib variant format that Emmanuel Marty implemented in his APULTRA compressor. DX0 decompresses about 25% slower than the latest version of my LZSA2 decompressor (which I have not uploaded here, yet), but it is a lot smaller, at 193 bytes vs. 256 bytes. Personally, I suspect that LZSA2 or LZSA1 are currently the best choices for most current homebrew development situations, but ZX0 is certainly a project to watch. The biggest current problem with ZX0 is that Einar's compressor is *horribly* slow, but hopefully that will improve in the future when some more experienced developers take a look at creating their own compressors. Anyway, here is the PCEAS source for my ZX0 decompressor, which uses some self-modifying code, and so is not suitable for a HuCard ROM, but the self-modifying code can easily be removed at the cost of a few % in decompression speed. zx0_6280.s (6 KB)
|
|
touko
Punkic Cyborg
Posts: 106
|
Post by touko on Apr 7, 2021 8:14:59 GMT
Thanks for sharing
|
|
|
Post by DarkKobold on Apr 14, 2021 22:06:27 GMT
So, the big problem with all of these is that you'd essentially need to write your own compressor.
When HuC takes a PNG or PCX, it converts it into bitplane format before putting it into the ROM. It's also smart enough to treat tiles and sprites differently. To use any of these decompression algorithms in code, you'd want them to decompress into bitplane format, so they'd need to be compressed that way.
Speed and compression, imo, is 2nd to usability. I don't know if it's better to have a utility that chooses a compression format and goes with it, or just a utility that spits out a ".bin" file, which is just the bitplane information from a png, and then you compress that.
|
|
|
Post by elmer on Apr 17, 2021 1:21:11 GMT
So, the big problem with all of these is that you'd essentially need to write your own compressor. Why is that? There are already compressors written for these formats, and the source is available to modify, if you need some kind of custom output format. When HuC takes a PNG or PCX, it converts it into bitplane format before putting it into the ROM. It's also smart enough to treat tiles and sprites differently. To use any of these decompression algorithms in code, you'd want them to decompress into bitplane format, so they'd need to be compressed that way. Yes, that is generally correct, although it is *possible* to convert from packed-pixels to bitplanes during (or after) decompression if you don't care about the speed penalty. I don't know if it's better to have a utility that chooses a compression format and goes with it, or just a utility that spits out a ".bin" file, which is just the bitplane information from a png, and then you compress that. The second option is preferable, and it is something that we have talked about in previous threads. Putting together your own conversion toolchain, rather than just relying on what is built into HuC/PCEAS, will open up vistas of options and capabilities that you just don't have now, as you are currently relying on the converters and library code that other people have written. There is nothing wrong with using the existing libraries, and there is a whole universe of Unity developers that can't program their own low-level code, but who can still produce "product". But until the time that you do choose to take the step of moving to a custom toolchain outside of the converters built into HuC/PCEAS, then "no", I'm afraid that you are unlikely to find the topics in this thread to be of much use to you. FYI, I did take a look at adding compression into PCEAS, but gave up, because PCEAS really doesn't like the size of sprite/backround data to change between passes. Perhaps turboxray will decide to have a go at adding the capability to PCEAS ... but even if he does, it is still better (IMHO) to do the job with a separate tool and to then just incbin the final result, or to include the data as a file in ISOLINK, because you (as the developer) have more options that way.
|
|
|
Post by DarkKobold on Apr 17, 2021 2:04:28 GMT
So, the big problem with all of these is that you'd essentially need to write your own compressor. Why is that? There are already compressors written for these formats, and the source is available to modify, if you need some kind of custom output format. When HuC takes a PNG or PCX, it converts it into bitplane format before putting it into the ROM. It's also smart enough to treat tiles and sprites differently. To use any of these decompression algorithms in code, you'd want them to decompress into bitplane format, so they'd need to be compressed that way. Yes, that is generally correct, although it is *possible* to convert from packed-pixels to bitplanes during (or after) decompression if you don't care about the speed penalty. I don't know if it's better to have a utility that chooses a compression format and goes with it, or just a utility that spits out a ".bin" file, which is just the bitplane information from a png, and then you compress that. The second option is preferable, and it is something that we have talked about in previous threads. Putting together your own conversion toolchain, rather than just relying on what is built into HuC/PCEAS, will open up vistas of options and capabilities that you just don't have now, as you are currently relying on the converters and library code that other people have written. There is nothing wrong with using the existing libraries, and there is a whole universe of Unity developers that can't program their own low-level code, but who can still produce "product". But until the time that you do choose to take the step of moving to a custom toolchain outside of the converters built into HuC/PCEAS, then "no", I'm afraid that you are unlikely to find the topics in this thread to be of much use to you. FYI, I did take a look at adding compression into PCEAS, but gave up, because PCEAS really doesn't like the size of sprite/backround data to change between passes. Perhaps turboxray will decide to have a go at adding the capability to PCEAS ... but even if he does, it is still better (IMHO) to do the job with a separate tool and to then just incbin the final result, or to include the data as a file in ISOLINK, because you (as the developer) have more options that way.
Maybe "write your own compressor" was the wrong wording. I mean write your own wrapper, as we've both outlined.
I'm using custom toolchains already for other things to do "incbin" so having a few extra executables that run before my code compiles is not a major step, right now. I mostly use them for dumping excess data from promotion maps for SGDK/GBDK, since those don't have native support for STM. Also GBDK splits it into banks for me, to save some effort.
I'd still probably need help with that exact step though - just an exe that takes a png, you specify if it's a tile or sprite, and then outputs the bitplane format. Ideally that exe would also do the compression to save some effort.
|
|
|
Post by gredler on Apr 20, 2021 22:51:02 GMT
The code seems a bit more complex than the LZ4 one. Most of what looks like complexity in my LZSA decompressors is just optional code paths so that smaller-or-faster versions of the code are both in the same source file. You can look at Peter Ferrie's 6502 versions of the code in the LZSA distribution on gihub. They're easier to read, but decompress significantly slower than mine. He's the guy that wrote the 6502 LZ4 decompressor that you're probably looking at. Linking Peter's version of the code for convenience. - github.com/peterferrie/lzsaI am again looking at this from a art production standpoint and hoping we can get something going to save some space . That budget is getting more daunting every time I look at where we are and where we need to be hahah
|
|
TailChao
Gun-headed
I Must Eat Muffin Gear.
Posts: 68
Fave PCE Game Overall: Bonk's Adventure
|
Post by TailChao on Apr 22, 2021 13:49:26 GMT
I did take a look at adding compression into PCEAS, but gave up, because PCEAS really doesn't like the size of sprite/backround data to change between passes. I'm still off in WLA-DX land, but I vote for keeping any of the data conversion or compression steps out of the assembler. It can make the build process hideously convoluted if your packed assets call out to code, then the addresses change on subsequent passes, which affects the packed size and - yergh! There's some merit in just statically allocating a few banks for compressed assets and a directory, then running your build process twice
|
|
|
Post by elmer on Nov 6, 2021 15:26:33 GMT
The biggest current problem with ZX0 is that Einar's compressor is *horribly* slow, but hopefully that will improve in the future when some more experienced developers take a look at creating their own compressors. Woohoo ... the amazing Emmanuel Marty has created a new compressor for ZX0 called Salvador. Some quick testing shows that it turns ZX0 into something that is (IMHO) usable for development. File Size Compressor Time ========================================== PoPCore.bin 40,164 PoPCore.puc 24,368 pucrunch -c0 -d 0.5s PoPCore.lz2 23,922 lzsa.exe -f2 1.0s PoPCore.zx0 23,310 zx0.exe -quick 2.5s PoPCore.zx0 22,655 salvador.exe 1.0s PoPCore.zx0 22,646 zx0.exe 35.5s There's some merit in just statically allocating a few banks for compressed assets and a directory, then running your build process twice Yep, this is basically what PCE CD developers did, especially later on. Seiya Monogatari and both Legend of Xanadu games, all have core sections of game code that doesn't change, and then they swap between different 192KB (or larger) files with lots of different chunks of compressed data inside them. The games dedicate an 8KB bank for decompressing sprite/background/map assets as needed, and then either uploading the assets to VRAM, or using them once decompressed. After uploading art assets to VRAM, that dedicated 8KB bank is used for running decompressed scripts, so no memory is wasted. Variations on this basic kind of scheme have been used in professional game development for decades.
|
|