QT Decode JSON layer data

Questions
1

Hi,

I’m trying to build an engine around the JSON exported data.
I am using QT, but i’m having difficulty reading data. Using old versions in the past, the data was not base64. I’m building around chunk based data but during testing, I can’t convert the base64 data to a usable state.

This is an example code

QFile file(":NinjaRPG/NinjaRPG.json");
    if (!file.open(QIODevice::ReadOnly | QIODevice::Text)) return;
    QJsonDocument doc = QJsonDocument::fromJson(file.readAll());
    file.close();

    if (!doc.isObject()) throw "An error has occured";
    m_jsmap = doc.object();;
    QJsonArray layers = m_jsmap["layers"].toArray();
    for (int i = 0; i < layers.size(); ++i) {
        QJsonArray chunks = layers[i].toObject()["chunks"].toArray();
        for (int b = 0; b < chunks.size(); ++b) {
            QJsonObject chunk = chunks[b].toObject();
            QByteArray arr = chunk["data"].toString().toUtf8();
            qDebug() << arr;
        }

    }

I am trying to get the data to then filter through GIDS. The problem is the decoding i can’t seem to grasp.

EDIT: The data return is not decoded, I’ve tried using .data() however qDebug should show this without a call to it.
I’ve searched around and found tileson’s C++ library but would prefer to build from scratch for full control…

2

I’m not familiar with Qt or with infinite map chunk parsing, but you can set the map to use CSV encoding in Tiled if you want to avoid base64 and compression. This makes for much larger files though! I don’t recommend using CSV for larger maps in production.

If you do want to use compression for the size benefits, then you’ll need to decode the chunk data (base64 decode, followed by a decompression based on whatever compression is set on the layers object in the JSON), then you need to look at two properties on the layers object in the JSON:

  1. encoding, which will be either csv for CSV, or base64 for base64.
  2. compression, which will tell you which compression method is used.

You can assume both of these if your input files are consistent, but for a robust parser, I recommend implementing the ability to parse all of these.

The chunk[b].data is the encoded part. It’s not enough to toString().toUutf8() it, because it’s already a string - a base64-encoded string. You’ll need to use a base64 library (which I think Qt includes) to decode it, e.g.

QByteArray chunkData = QByteArray::fromBase64(chunks[b].data);

At this point you’ll have a byte array of the compressed data. You’ll need an appropriate library. I believe Qt has qUncompress(const QByteArray &data) for zlib, and there are implementations available for gzip and other methods. Uncompress it and you’ll have a new QByteArray that will contain the raw data. However, you need an array of unsigned ints rather than bytes, so you’ll need to convert or memcpy this byte array to the appropriate type. Once you’ve done that, you can access the GIDs in that array.

1 Like
3

This is where I’ve been having issues. I’ve saved the exported file with no compression to avoid adding complexity to my test code. Chunks are basically small images 32x32 tiles with a position that can be rendered and merged together at runtime. It’s an array of maps in other words with an x,y position.

Chunks are stitched together at runtime. Minecraft is a great example of this but chunks are three-dimensional versus two.

chunks[b].toObject() //returns the chunk object, includes x,y,witdh,length and data of that chunk

I’ve gone through QT’s documentation on QByteArray but no creator or cast allows me to define a QByteArray from a QString, the result of reading the object of chunk[b].toObject()

fromBase64 is parsed from a QByteArray.

Since the Base64 string imported for that chunk is uncompressed, I should be able to simply decode it through QByteArray I believe. This is where I have no success. Is it that QByteArray cannot decode the base64 encoded data? base64 is a standard thus I have to be missing something.

For reference, this is an output of one of the test chunks that I’m attempting to decode

"QQAAAEIAAABBAAAAQgAAAEEAAABCAAAALQEAAC4BAAAvAQAAAAAAAAAAAAAAAAAARgAAAAAAAAAAAAAAAAAAAEEAAABbAAAAXAAAAAAAAAAAAAAAWwAAAFwAAABdAAAASQAAAAAAAAAAAAAAAAAAAEYAAAAAAAAAAAAAAAAAAABWAQAAcAAAAJUBAACWAQAAAAAAAHAAAABxAAAAcgAAAAAAAAAAAAAAAAAAAAAAAABGAAAAAAAAAAAAAAAAAAAAVgEAAHAAAACqAQAAUwEAAFQBAACFAAAAhgAAAIcAAAAAAAAAAAAAAAAAAAAAAAAARgAAAAAAAAAAAAAAAAAAAFYBAABwAAAAZwEAAGgBAABpAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEYAAAAAAAAAAAAAAAAAAABWAQAAcAAAAHwBAAB9AQAAfgEAAFYBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABGAAAAAAAAAAAAAAAAAAAAVgEAAHAAAACRAQAAkgEAAJMBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABwAAAApgEAAKcBAACoAQAAXgEAAF8BAABgAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcAAAAGEAAABiAAAAYwAAAHMBAAB0AQAAdQEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASgAAAHAAAAB2AAAAZAAAAHgAAACIAQAAiQEAAIoBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAdgAAAGQAAAB6AAAAYgAAAGIAAABiAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHkAAABkAAAAZAAAAGQAAABkAAAAZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkAAAAZAAAAGQAAABkAAAAZAAAAGQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZAAAAGQAAABkAAAAZAAAAGQAAABkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABbAAAAXAAAAF0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcAAAAHEAAAByAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="

I’m stuck with this at the moment, the rest of the implementation including stitching rendered grids I’ve done a few times :confused:

EDIT: As mentioned, I want to avoid using CSV, and plan to compress the data in the future, this as specified is easy to decompress with QT’s built in functions. Though I need to parse decompressed data first :frowning:

As an added reference, Tiled is based off of QT, so the fact QByteArray isn’t acting as I’m trying to use it, i.e from utf8, frombase64, etc, has got me stuck…

@eishiya If i comprehend your last statement, since the data is already not compressed, i still need to convert it to unsigned ints? Is this why my base64 unencoded data from QByteArray remains the same?

Thank you very much for your answer earlier, trying to get my head around base64 encoding.

4

You should not be converting the byte array to ints until after you’ve decoded and decompressed it. Until then, it’s not int data.

QString::toUtf8() returns a QByteArray, so that’s the “conversion” method you should probably use to get a QByteArray from data that you can then pass into QByteArray::fromBase64(), which in turn should get you the decoded (but still compressed) byte array, which you can then decompress with the zlib or whatever other functions.

Sorry if I’m stating the obvious, I don’t have much context here.

5

Just two things to add, since @eishiya already pointed out the main thing missing in the original code snippet, which is a call to QByteArray::fromBase64 to actually decode the base64 data:

  • You could use QString::toLatin1 instead of QString::toUtf8 since you already know it’s base64-encoded data, so there shouldn’t be any characters outside of the Latin1 range. Might be a bit faster.

  • The previously mentioned qUncompress can only uncompress data that was previously compressed using qCompress, due to its custom 4-byte header. Tiled uses zlib directly instead (also since it needs to support gzip as well), but alternatively one could prepend the custom header as mentioned in the documentation.