Improper colors

[turboxray]

Some more examples: release_candidate_1 left, straight RGB on the right:





The extra shade in the cloth, but the shadow in the lower breast plate is less red-magenta glow-y (i.e. it's darker in the non RGB one, making it more natural/correct). I'm thinking posting pics like this is might be kinda useless. My iphone and macbooks show much more difference than three different PC setups, for the grey shade in the cloth. At least the kiaidan one seems to be the same level of difference between all my devices.

[turboxray]

dup

[tatsuya79]

Nice, it's a lot about the blue tints.

I can see it fine on my PC with a good VA monitor.

Ah yeah... if it's about the grey it's really subtle, lowering the gamma level makes it more apparent.

[turboxray]

Jul 12, 2020 9:07:51 GMT tatsuya79 said:

Nice, it's a lot about the blue tints.

I can see it fine on my PC with a good VA monitor.

Ah yeah... if it's about the grey it's really subtle, lowering the gamma level makes it more apparent.

 Blues are definitely noticeable. Skin tones are more natural and less orange/yellow on the real VCE. Reds, Greens, Blues in the very low/dark end blend better (in RGB, they're too over saturated for how dark they are - making them stand out when they shouldn't). 

[tatsuya79]

I just replayed Ankoku Densetsu, it's a lot more ankoku (dark).

It fits the mood a lot better, I always thought the colour choices were strange...

[tatsuya79]

Another interesting difference with Avenger.

[tatsuya79]

Jul 9, 2020 2:25:43 GMT dshadoff said:

...But there's another layer down the rabbit hole. We also saw on the Startling Odyssey II overworld (just before entering battle) that the grass has addiitonal colours when shown on a composite monitor than are shown when displaying using just the composite palette. The specific colour-transitions in the grass cause NTSC artifacting, which is used to great effect on this grass scene... points of darker colours which don't actually exist in the source material appear as part of the image, and complement it well. The artists definitely planned this aspect.

We don't yet have a solution for that though... but we might start looking into it in the not-too-distant future.

I tried to see if that effect could be recreated with some shader/filtering on top and it seems to work fine.

crt shader:

crt shader + blargg composite:

...I could have boosted the gamma or brightness a bit on that last pic.

edit: gamma boost and higher blargg res to regain some details:

[dshadoff]

Interesting. I haven’t seen the blargg composite filter. What are you plugging it into, and where did you get it ? (Is this mednafen ?)

While it doesn’t look like a complete match, it certainly goes part of the way.

[tatsuya79]

It's not exactly as it look in that pic, as blargg filter is refreshing the artifacts every frame, so you see a bit more on your screen.

I'm using beetle_pce in retroarch + crt geom shader (boosting the luminance) + blargg composite filter as I feel it gives a better result than most ntsc composite shaders, but your preference could differ.

I added the wip palette into the core (it's a mednafen fork if you don't know) so it can be changed directly in the core options (and saved as a game override).

Tell me if you want anything changed in the credits there btw (or you could submit a change yourself too as everything is open source).

[dshadoff]

I was just wondering what you were looking at, and whether it would be easy to set it up here (and whether the blargg stuff could easily transpose to MiSTer).

As for credit, I think turboxray, furrtek and others deserve it more than I do.
turboxray brought it to my attention and I just got some interested people together to work on it.

[tatsuya79]

Dungeon Explorer II does some composite magic as well.

 

[samiam]

If anyone here has an oscilloscope and a PCE with composite output, I would be very grateful if you could take a measurement for me.

What I'd like is for you to look at the difference between blanking level (which is the same as black in Japanese NTSC and in all PCE/TG16 revisions) and 100% white in the composite video signal (with 75 ohm termination). The 240p test suite has a convenient full-white screen that works well for this, but any game with a large, stable section of pure white will probably suffice.

In the NTSC standard, 100% white should be 714mV above blanking level. A few of the folks looking into creating new palettes for use in emulation, including myself, have looked to see how close real PCE hardware is to the standard, and what we've found is a startling level of variability.

The data is still a bit thin, but it appears that stand-alone systems with built-in composite (such as the Coregrafx models, the Supergrafx, the Shuttle, etc.) generally fall around 650mV +/- 20mV. Meanwhile, two TG16 CD docks have measured an alarmingly low 558 and 564mV on 100% white, and an AV Booster paired with a few different base PCE systems turned out 736-760mV.

It would be really good to have a few measurements for each PCE console revision. No matter what system you might have, the more measurements the merrier. The goal is to establish what a good "reference" PCE might be, or whether such a thing arguably exists.

Thank you! 

[elmer]

Jun 30, 2020 14:45:40 GMT dshadoff said:

What I can share is a Mednafen palette file (NOT FINAL !) and a couple of snapshots.
...
This isn't the final version of the palette, but I think it's the best one yet.

How is the job going of figuring out the palette?

I see that turboxray has mentioned that something was found out about a non-linear R-Y and B-Y conversion, and I see that Mister now has a recent new palette that is marked as "final".

Is there a "final" palette available for Mednafen, and are y'all really at the "final" point?

[dshadoff]

Wow, good timing - I was just coming back to the board to see what project are in updatable status.

You have a MiSTer now ?  Or are you just watching the repositories ?

First, let me give a brief overview of what's happened in the past 2-3 months:
There was a pretty long process of determining actual colours through objective measurement - but measuring modulated signals is still a sort of an 'eyeball' type thing.
Artemio (of the 240p suite) has various vectorscopes to use with composite video, and struggled to get composite output from the MiSTer for comparison purposes.
Rysha (aka Kitrinx) worked on a pseudo-vectorscope application to do more objective measurements of the colour values of final palettes for comparison against the vectorscope captures.

We found a few non-linearities in output of the PC Engine video - these seem to arise from the amplifier stage.
And of course, a vectorscope only measures colour - not luminosity - so, another dimension needed to be added for measurement.

And to top it all off, we found that different units (from among SuperGrafx, Duo, Duo-R, several CoreGrafx) produced slightly different outputs, with respect to overall level and colour rotation on the vectorscope.  But several key features of the reading seemed to remain consistent.

In the end, after a series of trial-and-error adjustments, Rysha came out with a palette which matched on the pseudo-vectorscope, had similar output levels, and looked accurate to the eye.  (There were many iterations to get to that point).

She has committed to get this onto a GitHub repo, but it hasn't appeared just yet.
The vectorscope should also be available soon too (probably in its own repo).

She published this snippet of code on Artemio's discord channel - I don't recall whether this was just before or just after she made one final adjustment, but here you can see the math that goes on in calculating the palettes:

(Note: It will generate a mednafen palette as one of its outputs)



#include <stdio.h>
#include <stdint.h>
#include <stdbool.h>
#include <math.h>

#include "bfbii.h"

#define LODEPNG_COMPILE_ENCODER
#define LODEPNG_COMPILE_DECODER
#define LODEPNG_COMPILE_DISK
#include "lodepng.h"
#include "lodepng.cpp"

#include "color_lib.h"

#define SIDE_Y 512
#define SIDE_X SIDE_Y


void yuv2rgb(double y, uint8_t cb, uint8_t cr, uint8_t *r, uint8_t *g, uint8_t *b)
{
double r1, b1, g1;

// Table with the 1's complement translation of chroma
double c_table[32] = {
-15, -14, -13, -12, -11, -10, -9, -8,
-7,  -6,  -5,  -4,  -3,  -2,  -1, -0,
0,   1,   2,   3,   4,   5,   6,  7,
8,   9,   10,  11,  12,  13, 14,  15
};

// Y is 0 to 31, with standard ntsc-j range applied.
y = (y / 31.0) * (235.0/255.0);

// Populate color coefficients to NTSC REC 601
struct color_coeff co = {0};
pop_rec_const(REC_601, &co);

// Normalize to MAX to -MAX for each signal. 14/15 roughly represents the standard
// reduction constants (235/255) that would be applicable to these signals. Since the
// table never goes below 1 or above 30 It's assumed this was intentional.

double pb = ((double) c_table[cb] / (c_table[31])) * co.u_max;
double pr = ((double) c_table[cr] / (c_table[31])) * co.v_max;

// The original signal was likely reduced directly from normalized U and V values to the
// ntsc reduced values with no intermediate u or v max values, however as this was simply
// reversed by the decoder, there's no need to introduce additional complexity or quantization
// errors by doing so here.

// Apply phase shift. This appears to vary a bit per system, but it definitely exists
// intentionally as without it, the colors would be non-sensical. This was based on
// a DUO-R belonging to Artemio.

double phase_deg = -23.5; // in degrees
double rad = deg_to_rad(phase_deg);
double s = sin(rad);
double c = cos(rad);
double old_pb = pb;
pb = pb * c - pr * s;

phase_deg = -9.5; // in degrees
rad = deg_to_rad(phase_deg);
s = sin(rad);
c = cos(rad);
pr = pr * c + old_pb * s;

// The combination of the rotation above ends up adding up to -33 degrees.
// I don't know if it's a coincidence or somehow related to the 33 degree rotation of
// YIQ color model, but my guess is coincidence.

// Reference NTSC YUV to RGB conversion.
yuv_to_rgb(&co, y, pb, pr, &r1, &g1, &b1);

// Create 8 bit colors

// Expand the range of colors from limited to full, as a television would have done.
double m = (255.0/235.0) * 255.0;
//double m = 255.0;
r1 *= m;
g1 *= m;
b1 *= m;

// double contrast = 30.0;
// double F = (259.0 * (255.0 + contrast)) / (255.0 * (259.0 - contrast));
// r1 = (F * ((r1 + 16) - 128.0)) + 128.0;
// g1 = (F * ((g1 + 16) - 128.0)) + 128.0;
// b1 = (F * ((b1 + 16) - 128.0)) + 128.0;

// Clip where needed
r1 = r1 > 255 ? 255 : r1 < 0 ? 0 : r1;
g1 = g1 > 255 ? 255 : g1 < 0 ? 0 : g1;
b1 = b1 > 255 ? 255 : b1 < 0 ? 0 : b1;

*r = round(r1);
*g = round(g1);
*b = round(b1);
}


int main (int *argc, char *argv)
{
uint8_t rgb_lut[512][3] = {0};

uint8_t r, g, b;

for (int32_t x = 0; x < 512; x++) {
uint16_t index = ((index_lut[x][0] << 6) | (index_lut[x][1] << 3) | (index_lut[x][2])) & 0x1FF;
int8_t y, u, v;
yuv2rgb(pbpry_int_lut[x][2], pbpry_int_lut[x][0], pbpry_int_lut[x][1], &r, &g, &b);
rgb_lut[index][0] = r;
rgb_lut[index][1] = g;
rgb_lut[index][2] = b;
}

unsigned char *raw_image = calloc(1024 * 1024 * 3, 1);

for (int32_t x = 0; x < 512; x++) {
printf("%3d: (%3d %3d %3d)\n", x,
rgb_lut[x][0], rgb_lut[x][1], rgb_lut[x][2]
);
// Draw the palettes
size_t ind = x * SIDE_X * 3;
for (int y = 0; y < (SIDE_X * 3); y+=3) {
raw_image[ind + y] = rgb_lut[x][0];
raw_image[ind + y + 1] = rgb_lut[x][1];
raw_image[ind + y + 2] = rgb_lut[x][2];
}
}

// Make the files
unsigned char *pngfile = NULL;
size_t pngsize;

lodepng_encode_memory(&pngfile, &pngsize, raw_image, SIDE_X, SIDE_Y, 2, 8);

FILE *f = fopen("input1.png", "wb");
fwrite(pngfile, 1, pngsize, f);
fclose(f);

free(raw_image);
free(pngfile);

// MiSTer Palette
f = fopen("palette.tgp", "wb");
for (int32_t x = 0; x < 512; x++) {
uint16_t rgb[3] = {0,0,0};

rgb[0] = rgb_lut[x][0];
rgb[1] = rgb_lut[x][1];
rgb[2] = rgb_lut[x][2];
fwrite(rgb, 1, 3 * sizeof(uint16_t), f);
}
fclose(f);

// Mednafen Palette
f = fopen("palette.pal", "wb");
for (int32_t x = 0; x < 512; x++) {
uint8_t rgb[3] = {0,0,0};

rgb[0] = rgb_lut[x][0];
rgb[1] = rgb_lut[x][1];
rgb[2] = rgb_lut[x][2];
fwrite(rgb, 1, 3 * sizeof(uint8_t), f);
}
fclose(f);
}


Full source here:
tg16color_Kitrinx.7z (156.79 KB)


As for the question of "final"... I think we are. There are still measurements which would be nice to have, but they are merely expected to confirm what we believe.

[elmer]

Sept 26, 2020 15:00:19 GMT dshadoff said:

You have a MiSTer now ?  Or are you just watching the repositories ?

First, let me give a brief overview of what's happened in the past 2-3 months:

...

As for the question of "final"... I think we are. There are still measurements which would be nice to have, but they are merely expected to confirm what we believe.

Nope, I don't have a MiSTer, I just saw turboxray posting about the new palette on AtariAge and so took a look at the MiSTer repository.

Thanks for the update!

I look forward to the new palette appearing in some future Mednafen update, but even before that I'd like to try it so that I can see if my small tweeks to LoX UI colors need more tweeks, or if they should just be entirely removed.